soc

As I have done in previous years, I am again participating in the Google Summer of Code as mentor for the Samba project.

Last year I Andrew and I co-mentored three students with mixed results. In the end we had to drop one of our students but the other two did well. I've only taken on one student this year for various reasons.

The amount of time required to mentor a student varies wildly depending on the student and is hard to predict based on their application. Some students seem to require quite a lot of mentoring while others are self-motivated and self-learning. This has not just been my experience, I've heard similar stories from fellow mentors on other projects.

Last summer Ricardo worked on SWAT for Samba 4 and he is still actively working on the project, even after the Summer of Code has finished. I hope to find the time to package SWAT in time for Debian Squeeze. At the moment SWAT just supports managing shares but Ricardo is working on user management.

In 2009 Calin worked on the GTK+ frontends for Samba, in particular changing them to be Python-based rather than C-based. This year his work is going to be continued by Sergio, hopefully with the some user-ready tools as the end result.

cp: Gazpacho - 117

These past few days in the US were a bit of a rollercoaster. Some random observations:

• The mentor summit was very nice and well organized (or rather: well disorganized). Lots of awesome people around from a wide variety of projects and nationalities.
• "Next Generation VCS" seems to be an alias for git these days in the minds of most people.
• I didn't write a single line of code in almost a week, something that is very rare.
• Driving an automatic gives you two spare limbs to use for other things. What those other things are, I have yet to figure out.
• Is the fact that your kid was student of the month or the fact that you own two cats and a dog really something that belongs on a bumper sticker?
• Gas is cheap (compared to Europe). I drove 300 miles on a $30 tank.
• The malls in the Bay Area are some of the biggest I've ever seen, but strangely enough they seem to lack both book- and cd-stores.
• Visiting Fry's continues to have a significant effect on the contents of my wallet.
• It is legal to turn right on a red traffic sign in California unless otherwise indicated. It took me a while to realize this until people repeatedly started honking behind me...
• The waiver I had to sign to be able to skydive in California was scary. I can cope with my operating system coming without even the implied warranty of merchantability or fitness for a particular purpose, but my parachute?
• I stopped pretending to have any regularity in my sleeping habits. 6 AM flights? It seemed like a good idea at the time.

For this years (the fifth?) Summer of Code, I participated once again as a mentor for the Samba and OpenChange projects.

Samba was assigned four slots this year: one was a CIFSFS project mentored by Steve French and the other three were Python projects related to Samba 4, co-mentored by Andrew and me. Our students did very well this year, although we unfortunately had to drop one after the mid-term evaluations due to lack of effort. Nonetheless, we're very happy with the results of the other two projects:

Calin Crisan (France) converted the rest of the applications in SambaGtk to Python, and worked on a GTK+ user manager for Samba and Windows. With his improvements, it is now possible to edit registries, manage users, inspect the endpoint mapper, plan tasks and manage services on a remote Windows machine using a GTK+ application on a Linux workstation.

Ricardo Velhote (Portugal) designed and implemented a new version of SWAT - the Samba Web Administration Tool. Unlike the old SWAT, his implementation is more than just a simple web-based editor for smb.conf. As we were expecting at the start of the Summer of Code, not all of the functionality could be implemented properly in a couple of months, not while getting the design and infrastructure right. With a basic version working, we now hope the remaining subsystems can be contributed with help from the community.

I'm planning to merge Calin's improvements to Samba-Gtk into the mainline in the next month or so. SWAT is a standalone application and will continue to live as a separate project, while being a part of the Samba ecosystem. Congratulations to both Calin and Ricardo on their achievements!

Google's Summer of Code was again a pleasure to participate in! I'm quite pleased with the overall results and have enjoyed working on the CUPS tree. Mike, main author of CUPS, has merged my Kerberos branch into trunk a couple of days ago. It should be part of the upcoming 1.3 series. My aim is to stick around and maybe work on SPNEGO/NTLMSSP support.

As of recently I have started working part-time for SerNet. I'll be working on random Samba bits for them, all free software. The first thing I'm working on is backporting the excellent libndr (one of the core parts of DCE/RPC) subsystem from Samba 4 to Samba 3.

The university year has also started, so my activity in free software will probably decrease a bit in a few days.

After cycling and playing cards in the Vosges (France) I'm back working. After browsing through 10 days of mail, I'm now making significant progress on CUPS-krb5 again.

Currently Playing: Rory Gallagher - Bad Penny

I'll be participating in the Google Summer of Code again this year. My project this time is Kerberos integration in CUPS.

My Debian application also finally got moving, after a little less than half a year. Hopefully I can get through this quickly.

Currently Playing: Counting Crows - Goodnight Elizabeth

It's almost 2 AM here, in other words, 17:00 PM in California - just a few more minutes before the list of SoC applicants will apparently be published. Hopefully one of my applications will be accepted; the quality of applications appears to be better overall than last year, so it'll be tough...

Currently Playing: Stevie Ray Vaughn - Crossfire

Google has posted a summary of the results of the Summer of Code on the Company Blog. There's also a link in there to a cool map with all the participants and mentors.

[Update 23:13] Slashdot now also has a story.

So, just a few more minutes until the deadline of the Google Summer of Code, 2005! I got pretty much all that I wanted in there done, except for the ldb_map module, which is still a bit dodgy.

It sure was nice and interesting to do participate! The other participants that were working on Samba for SoC have produced some great code and it looks like they're going to stick around. Especially Brad's work on DSSYNC looks promising. It'd certainly recommend anyone to participate if Google is holding another SoC again sometime in the future!

Looking back, I guess it could've been interesting to work on a different project then Samba such as mono (I do actually actively follow mono development, I just don't contribute at the moment), althought what I did now was probably more productive and allowed me to spend some time on other Samba things as well :-)

I just committed the following file 'STATUS' to the tree:

--- Samba3 -> Samba4 Upgrade ---
(C) 2005 Jelmer Vernooij <jelmer@samba.org>
Published under the GNU GPL

Sponsored by the Google Summer of Code program (http://code.google.com/summerofcode.html)
Mentored by Andrew Bartlett <abartlet@samba.org>
Thanks!

Done:
- Reading wins.dat
- Reading registry.tdb
- Reading passdb.tdb
- Reading account_policy.tdb
- Reading group_mappings.tdb
- Reading winbindd_idmap.tdb
- Reading share_info.tdb
- Reading secrets.tdb
- Reading smbpasswd
- Reading + writing (generic) smb.conf files
- Testsuite for read support mentioned above
- Console utility for dumping Samba information
- Import user accounts in Samba4
- Import groups in Samba4
- Import secrets in Samba4
- Import WINS data in Samba4
- Dump idmap data to LDB
- Import registry keys/values in Samba4
- Import account policies in Samba4
- Testsuite for upgrade
- Console utility from upgrading from Samba3 -> Samba4
- SWAT (Web interface) support for upgrading from Samba3 -> Samba4
- LDB generic mapping module
- (Experimental) Samba4 LDB <-> Samba3 LDAP mapping module based on LDB generic mapping module
- Testsuite for Samba4 LDB <-> Samba3 LDAP mapping module

Source files:
source/lib/ldb/modules/ldb_map.c
source/lib/ldb/modules/ldb_map.h
source/lib/samba3/group.c
source/lib/samba3/idmap.c
source/lib/samba3/policy.c
source/lib/samba3/registry.c
source/lib/samba3/samba3.c
source/lib/samba3/secrets.c
source/lib/samba3/share_info.c
source/lib/samba3/smbpasswd.c
source/lib/samba3/tdbsam.c
source/lib/samba3/winsdb.c
source/lib/samba3/samba3.h
source/scripting/libjs/upgrade.js
source/scripting/ejs/smbcalls_param.c
source/scripting/ejs/smbcalls_samba3.c
source/param/generic.c
source/param/generic.h
testdata/samba3/verify
testprogs/ejs/samba3sam
source/setup/upgrade
source/scripting/bin/samba3dump
source/dsdb/samdb/ldb_modules/samba3sam.c
source/script/tests/test_s3upgrade.sh
swat/install/samba3.esp

Known remaining issues:
- [upgrade] Conversion from the smbpasswd/TDB passwords to ntPwdHash / lmPwdHash is broken. Couldn't find out why.
- [ldb_map] Conversion of attribute names in DN's is still a bit dodgy
- [ldb_map] mapped objectClass names may be mentioned multiple times in returned records
- [ldb_map] add/modify support not tested very well with LDAP yet (only LDB+TDB)
- [ldb_map] group membership is not yet mapped (only primaryGroupID / sambaPrimaryGroupSID)

Looking more closely at one of the other approaches I could've taken for migrating from Samba3. We will probably end up supporting this method later on anyway, once we expand the vampire code. I do think the path I've taken for SoC is the right one though - it allows the upgrade with the least hassle and user input.

This approach could be splitted up into the various parts:
- samba3 pidl backend
- Samba4 vampire + server side samsync support in Samba3
- unixinfo (\unixinfo) (exposes idmap)
- in Samba4 (client side)
- in Samba3 (server side)
- winsrepl (thru seperate pipe?)
- enum/add shares (\srvsvc)
- enum/add registry (\winreg)
- enum/add printers (\spoolss(?))
- convert smb.conf (using Jerry's registry hack)

Advantages of this approach:
- Can be used to 'clone' Samba4 boxes as well
- Relatively few extra loc necessary (a lot shared with vampire)
- 'Clean' code - simply enumerates and adds

Disadvantages:
- Needs running Samba3 daemon for upgrade (!)
- Needs (complex?) Samba3 code changes
- Needs latest version of Samba3

The one-way upgrade from Samba3 to Samba4 is now working, at least partially! There are both a web-frontend and a console utility that can do an upgrade, given a libdir (usually /var/lib/samba) and a configuration file (usually in /etc/samba/smb.conf).

Last night, I rewrote most of the conversion code to be in EJS rather then in C. One of the positive things about this is that I can use embedded LDIF inside of the JavaScript code and that I can build on the existing Samba4 provisioning code (which is in EJS as well). Writing EJS wrappers for my existing C code was easier then I had thought.

One of the other things I did today is add a fallback mechanism for ldb_map where it can store the data that could not be mapped. There are now two types of records in ldb_map: those that are partially mapped (mapped, but might have one or two attributes stored in the fallback database) and those that are completely stored in the fallback database.

Below are some screenshots of the upgrade procedure in SWAT:

<html><a href='/blog/uploads/screenshots/upgr1.png'><img width="110" height="83" border="0" hspace="5" align="left" src="/blog/uploads/screenshots/upgr1.serendipityThumb.png" alt="" /></a><a href='/blog/uploads/screenshots/upgr2.png'><img width="110" height="83" border="0" hspace="5" align="left" src="/blog/uploads/screenshots/upgr2.serendipityThumb.png" alt="" /></a><a href='/blog/uploads/screenshots/upgr3.png'><img width="110" height="83" border="0" hspace="5" align="left" src="/blog/uploads/screenshots/upgr3.serendipityThumb.png" alt="" /></a></html>

The sam.ldif and winbind.ldif generation code now works as well. I'll be working on generating the new smb.conf next.

One of the things that worries me is the fact that I can generate so few of the Samba4 'user' and 'group' attributes. Probably need to borrow some code from the current provisioning script to fix that up.

I'm wondering whether I should do some of the upgrade using EJS, so it can be integrated into
SWAT easily...

ldb_map and the samba3sam ldb module that is based on it work nicely now and appear to be memory-leak free. I have so far been testing with a TDB backend loaded with an LDIF from Samba3. Tomorrow will be the big day, testing against an OpenLDAP server.

Since the LDAP schema for Samba3 contains information that the Samba4 SAM database doesn't store and vice versa, it will be interesting to find out whether the current mapping will be sufficient. If it is not, I might have to fall back to storing the Samba4 attributes I can't store in the Samba3 LDAP database in a seperate database instead.

Another thing I did today is clean up the parameter list in Samba4 - I removed all unused parameters. This should hopefully make it easier to do the Samba3->Samba4 smb.conf conversion code.

TODO is down to:
- smb.conf conversion
- Finish sam.ldif / winbind.ldif from the TDB files
- Test samba3sam module against OpenLDAP

Today was a productive day. I just got the LDIF output for samba3dump working for wins.ldif and the various registry hives. TODO is down to:

- sam.ldif
- winbind.ldif
- smb.conf
- test and finish ldb_map and ldb_samba3

I hope to get the first three done tomorrow.

Next on my list is the generation of the LDIF's for the one-way upgrade path. These will be generated by internally building ldb_ldif structs and then dumping them using the LDB helper functions. This would easily allow us to later on directly write to LDB database.

I'll need to generate the following LDIF files:

- hklm / hkcu / etc.ldif: from samba3_regdb
- wins.ldif: from samba3_winsdb
- sam.ldif: from samba3_secrets, samba3_policy, samba3_privileges, samba3_samdb
- winbind.ldif: from samba3_winbind

I'll also make a start on the smb.conf output.

Finally got the ldb_map code done. This is a simple skeleton module that takes a mapping table of LDAP attributes and objectclasses and then maps between two different schema's. Also got the initial part of ldb_samba3 compiling, which is the first module that actually uses ldb_map.

Yesterday evening, I also added support for reading smb.conf files and share_info.tdb files to the lib/samba3/ directory. Looks like my TODO-list is now down to:

- testing (writing a couple of smbtorture tests, preferably)
- code for generating new smb.conf and for generating LDIF from the various TDB's

I've now finished the ability to read the various TDB files. I've also started on the "mapping" LDB module that will allow a simple mapping from one objectclass to another. We'll need this for the backwards compatible LDAP backend for Samba3.

This backend will never be used itself but instead have
other backends based on it (e.g. ldb_samba3 for a Samba3-LDAP-to-samdb
backend)

Each backend that uses this skeleton has to specify an array of
struct objectclass_mapping's (see ldb_map.h).

If no mappings are defined for an objectclass, it will be kept as is.

Will also keep cache of mapped dn's so we don't have to traverse the tree each time

So, TODO left:

- finish ldb_map
- fill ldb_samba3's mapping table
- support smb.conf parsing in lib/samba3/
- support generating LDIF's

My Summer of Code project is coming along nicely (although I haven't been very actively blogging about it). I'll be working on full-time until the deadline, I guess, but I think I'll make it.

So far I've got the following done:

- Reading of tdbsam
- Reading of policy database (still need to add privilege support)
- Reading of idmap database
- Reading of registry file (still needs some work)
- Reading of old smb.conf files
- Reading of group mapping database
- Initial work on ldb_map backend

The major lump of work now is the ldb_map backend and the conversion of the smb.conf.

After quite a bit of thinking and discussing on #samba-technical, I came to the conclusion that the following three approaches are possible:

1) TDB conversion approach. Read in TDB dump out LDIF (one-way)
- samr.ldb: from tdbsam/smbpasswd, account_policy.tdb, secrets.tdb, group_mapping.tdb, idmap.tdb, privilege.tdb
- registry.ldb: from registry.tdb
- wins.ldif: from wins.tdb/wins.dat
- smb.conf/ea's: generated from the old smb.conf + share_info.tdb

(one-way upgrades can be done by using ldbsearch -a on these dynamically
generated ldb's)
Since TDB's are local, there isn't much point in writing back backwards
compatible data.

2) samr "mapping" backend (alternative for samr.ldb) (two-way)
This would allow users to keep mixed domains containing Samba3 and Samba4.

3) The vampire way of doing things (one-way)
- samba3 pidl backend
- Samba4 vampire + server side samsync support in Samba3
- unixinfo (\unixinfo)
- in Samba4 (client side)
- in Samba3 (server side)
- winsrepl (thru seperate pipe?)
- enum/add shares (\srvsvc)
- enum/add registry (\winreg)
- enum/add printers (\winreg, perhaps also \spoolss(?))
- convert smb.conf (using Jerry's registry hack)

I will implement a combination of the first two - mainly because the third one would require vampire to be working better then it does at the moment, and I wouldn't be able to expect it to be ready before the end of summer. Furthermore, the third approach would require Samba3 to be working correctly at the moment of the upgrade.

This summer, I will be participating in the Google Summer of Code. The project I will be working on is the Samba3->Samba4 upgrade path.

In my original proposal, I splitted this project up into the following subtasks:

- Each of the various password database backends(tdb,ldap,smbpaswd) available in Samba3 need to be accessible and converted to the SAM database in Samba4.
- Printer database needs to be read from the Samba3 TDB and added to the apprioprate LDAP subtree.
- The WINS database needs to be converted from both plain text file and TDB file to LDAP/LDB.
- The account policy database needs to be read in from a TDB and applied to the data in LDAP/LDB.
- The privilege database needs to be read from a TDB and added to the SAM database in LDB.
- Group mappings need to be read from the TDB and added to the SAM database.
- The share info database and the configuration file from Samba3 need to be read and converted to either xattrs or a TDB in Samba4
- Secrets.tdb, containing the domain formation, needs to be merged into the SAM database.
- Last, but not least, a Samba3 configuration file should be parsable in Samba4. This file can be used for producing a
Samba4 smb.conf file as well as for updating the various databases mentioned above.

I will probably have to change this later on, but it's a good list to start with :-)