Many to Many with Data in the Join Table

When I wrote the last post, I had gotten nHibernate to create objects. They were specified to be in a many-to-many relationship and, though I can't remember the exact details of that moment, the objects were being created but the relationship was not. Since then, I have been really struggling to create that relationship.
The biggest thing that I have learned is that many-to-many is no longer fashionable and, I think, appropriately so. It is replaced by symmetrical one-to-many associations with a single relationship table. That is, a person visits many bars, and a bar has many visitors, but they really have a codependent relationship with each other. Which is to say, a person has many codependent relationships that help him be drunk and the bar has many codependent relationships that help it contribute to the delinquency of the culture.
For me, the nicest thing about this is that it removes the behind the scenes magic. It also makes use of that joining table. After all, it's not like there is some special kind of table for this. It's a table with data and making it a relationship rather than something invisible makes it more useful by giving us some semantics that allow us to store data about the relationship. The drunken person might want to note the number of blackout experiences enjoyed in each codependent relationship. The bar might want to record how many times police were involved for any particular codependency.
When I worked out how to use nHibernate <bag> elements and the <many-to-one> element and which entity they refer to (simple but slippery), I got an error message, "Cannot insert the value NULL into column." The column referred to is the foreign key in the join table. It's as if nH tries to create the join table first, before it knows the private keys of the other two records. Since I am using MSSQL, it allows you to specify whether a field is nullable and, nHibernate is really keen on using that when it generates a schema.

I communicated with the nice people on the google group (If you want to see the details, look at the technical conversation here) and tried many things. Inverse="true" did no good. Using <ibag> neither worked nor made sense. I tried using <set> just in case (couldn't figure out the type of the object property for that and it makes no sense because there can be duplicates and no sequence is implied). I tried using interlocking many-to-many. I tried changing the order of creation and saving in the app. Nothing mattered.

The only way I made any progress was to tell the database that the field is nullable but then, it complained that I was violating a "foreign key constraint". Sure enough, reading the generated schema, nHibernate adds this to the database, too. A foreign key constraint basically tells the database what two table/column pairs relate to each other and will not allow that to be screwed up. Good idea, unless you are using an ORM that creates the objects first and subsequently updates the foreign keys. I suspect that the update is when the violation occurs. I was far enough into it then that I knew there was no way I was going to convince nH to do it differently. I edited the schema again and Voila! Two concrete objects and a join table with correct foreign keys.

Tough luck on the database constraints.

Finally, as I write this, I am thinking about that guid. It's a long string and it's used as a key. I had chosen an nHibernate generator type that turned the guid into a string (uuid.hex) and can see that there is nothing that can be alphabetized about them. This is worrisome. I looked at the other guid generator schemes in nH (here) and found one that had a link to the rationale behind it. First, the article confirmed my suspicions. In simple tests, a guid is takes thirty times longer than an integer key. Big problem. This guys new guid generator makes for good, local commonality in the beginning characters. This reduces the penalty from thirty to .1, a 300x reduction. Unfortunately, using this method (guid.comb) gives me a type conversion violation in C#. I guess I have more work to do.

nHibernate is getting there

Phew! This is a hard thing to do.

I have done two major things lately. First, I made it so that I can generate database schema based on nHibernate mappings as well as generate update scripts that will revise the database to match changes in the nH mappings. This latter is one of the things I am most excited about. I can't tell you how annoying I have found it over the years to try to keep code and database schemas in sync. 

The other is that I have managed to write data using nHibernate. I haven't got all functions shaken down yet, but it's getting close.

The main trick for schemas was getting mapping to work. The nH docs are not very instructive, especially to a C# newbie. I had the devil of a time figuring out (and eventually just asked and got the answer; I wish I was rich enough to just pay someone so I didn't feel obligated to make a good faith effort to find out on my own) how to deal with object properties that are objects themselves. In this case, I had a property called name, that was a BaseName type with it's own properties of First, Last, etc. Turns out that the <component> element is the one to use. Once I heard that, the example of how to use it was for a Name object. 

The other thing that was hard was to deal with many-to-many. There are obvious provisions for this in the nH docs but it isn't really explained. The main thing is that I had to add a property to my class for the elements of the related collection. That is, if Widgets and Goodies are in a many-to-many relationship (maybe I should have chosen People and Bars, many People visit each Bar has many People visiting it) then you will, at some point need to assign one to the other, eg,

firstWidget.barList.add(someBar);
firstWidget.barList.add(anotherBar);

I had to learn how to create firstWidget.barList. This is probably easy for experienced C# people. For me, I had to started learning about enumerable types. List<Widget> (and List<Goodie>) work for schema generation. when I tried to write data, I got, "Unable to cast object of type." Ouch.

Turns out that nH, behind the scenes, changes the type of that grouping property (actually, I used the <bag> element) to its own kind of enumerable type. "List" is actually a type based on the interface, "IList". I don't thoroughly understand this typing business, but nH can't change out an actual type but it can assign a type to a property specified with an interface.

Bottom line, I used IList<Widget> and IList<Goodie> and Voila! I got me some database writing.

But there's more!

These are smaller things that I knew about, but that came to bite me anyway. First, because of the whole lazy loading thing, properties need to be "virtual". This was easy to debug because the error message was very explicit. In my case, I am working with an industry xsd-based type system so I got to use me some serious regular expression mojo to make it happen.

My text editor of choice is BBedit. I needed to change all the public properties from "public PropertyName" to "public virtual PropertyName". Unfortunately, the classes are public as well (public partial class BlahBlah{}) so I had to exclude the public classes from the substitution. Eventually I found someone that had constructed a conditional regex pattern that did the job:

     (?(?=^(the_only_string_you_dont_want_to_match)$)^$|.*)

Since it was constructed to deal with an entire line, I changed it to:

     public +(?(?=(partial)) |.*)

and replaced it with

     public virtual \1

I have run into the other major issue several times and so, when I hit it again, it was easy. The xml mapping files, class.hbm.xml, property needs its output directory to be set as an Embedded Resource. If they are, things work. If not, "No persister for ...".

In a closely related point, remember to set your xml file property Build Action to "Copy Always."

Onward!!

C# Controller Parameter

I am astonished to tell you that the incoming variable to a controller in a default ASP.NET MVC setup apparently must be named "id".

WTF?

Someone, please correct me on this. I just did a sequence of experiments that appear to prove it and it resurrects my innate distaste for MS technology.

mbUnit

In the process of shaking down nHibernate, I've learned a bunch about mbUnit. The main thing is that it really wants to be driven from it's management partner, Icarus. Test files will run using Visual Studio's built-in testing framework, but, you basically cannot see any output from the test except the pass/fail notation. That makes it very, very difficult to debug.

Icarus is a standalone program. You link it to the assembly dll for your testing project and it shows you the list of tests. When you run them, anything that you write to the console (Console.WriteLine();) comes out in the report as do normal sorts of runtime errors.

I'm happy to say that it is very good at seeing the inside of the target system that is being tested. nHibernate has been difficult. One of the things I didn't trust was that the tests were able to see all the files and nuances so I verified. Simply adding a reference to the target projectName.dll does wonders.

XSD to C#, Woo Hoo!

There is an industry standard data model made by some gigantic multi-year committee. When I learned about it, I got all excited. These people had designed my data structure!! Woo Hoo!

Then I discovered that it's - literally - 600 pages. Fortunately, there is an xsd file. Even better, Microsoft has a utility in its developer kit (a separate download, "Power Toys") called xsd.exe. It converts an xsd file into some sort of data object class definitions. Considering all the [attributes] they stack into this file, it must have some serious .NET mojo. I don't know how to work that yet. But, I grabbed it into my Macintosh, cleaned the gigantic amount of .NET cruft and converted it into an outline program so that I can organize the class definitions. 

My thought is to use the outliner format to select the subset of data elements I want to play with and then take it and convert it into javascript class definitions (ah, the wonders of bbedit and regular expressions). The same data definitions for both sides of my AJAX fence. That sounds like a great thing. I'm so excited.

nHibernate Works!

This has been among my most grueling experiences. Since Wednesday, nHibernate has been showing a "Security Exception" error message. I have been fighting it ever since with absolutely no progress. It has been brutal. I read and experimented, read more, tried something else. Did it again. Again. Again. Again. Again. Again. Weds and Thurs, I went home exhausted. I wrote on technical bulletin boards and got some great ideas from colleagues. There technical details are in this thread on the nhusers group on google. It basically started working for no real reason at the end of Friday. I do not know what fixed the problem. I care, but only a tiny, little bit. Working is working.

I have learned about trust levels in IIS7. There are five of them. The lower three will not allow debugging. The upper two give security exceptions to nHibernate unless it gets to late Friday afternoon and then they don't anymore. I also learned about the 'unblock' feature that prevents files loaded off the internet from being used inadvertently. Rather than asking you if you want to allow a web-sourced file to be used when the file is needed, Windows 7 requires you to select each file, right-click to Property and choose unblock. There is no option to cause it to propagate to enclosed files. Very inconvenient. Anyway, I had to do that for each of the nHibernate .dll files.

Things I got about nHibernate that were harder to find than they should have been:

You need to specify the assembly. I don't think there was a <mapping assembly="AssemblyName"/> attribute in the sample configuration file. It can be specified there or as a string parameter on an addAssembly() (or something like that) when the nHibernate session is created. I couldn't get the latter to work myself. 

This is common sense, but the data definition object class name has to either be the same as the name of the table (in a database with a name that matches the one in the database login string in .cfg). If the names are different for some reason, you can us a table attribute (<class name="AssemblyName.Services.BaseDataObjects.categorytBase, AssemblyName" table="categoryBase">) in class element in the mapping file (.hbm). 

I just went back, hours later, and took a look. It works all the way through to the database. Sweet.

nHibernate Emergency

I've spent the last two days trying to get my first bit of nHibernate 
to work. I am in "No Persister" hell. I have done everything and can't make it work.

Please, for the love of all that is holy, help me!! 

If you know anything about nHibernate, please visit this page at google groups, where I detail my pain and save my bacon.

mbUnit is on the Air!!

On my way into trouble for this iteration, I created a new project in Visual Studio intended to be the beginning of the real deal. During it's setup, it asked me what testing framework I wanted to use and offered me mbUnit. I chose it and went to work. I created my first bit of nHibernate (ouch!) and made a test to try it out. It failed and sucked up another horrifying amount of time and pain.

I found a small tutorial that just did a tautological unit test (assert a=a). One step explained that I should see the test in Test View. Nope. Nada. Another painful experience trying to make it work. Gave up. Wrote an inquiry to the mbUnit group on google. A nice guy got back pretty quick. He pointed me to a page that looked promising but was long. Since I was in the middle of something else (nHibernate, my new bete noir!), I tabled it for later on.

In my thrashing for nHibernate I, for obscure reasons, decided to make a new project for some reason. While doing it, I noticed that VS offered me, in addition to .NET MVC, an "mbUnit Project." OMG!! I said, Yes, and created one. It didn't look like anything special, so I continued on my Via Dolorosa until I ran out of steam there.

Hoping to salvage something from the day, I looked again at my mbUnit project. I wondered how it was going to know what I wanted to test since it's in an entirely different project from the test target. Let me digress briefly...

During this week's angst, I have come to understand much more about how .NET stuff works. I now know, among other things, where .dll files come from and what they do. They come from a Visual Studio build. I suspect that it is a combination compilation and link process. There is a directory called "bin" that, after I build, gets a filed like, ProjectName.dll. Get a load of that. To get nHibernate and mbUnit going, I had to add a "reference" to their .dll files. Very interesting.

Along the way, I found an explanation about nHibernate that suggested a better file organization (should have done it first but whatever I was learning from at the time got me distracted), ie, a directory specifically for third party software. This seemed like a good enough idea that I paused in my efforts to move things around. That caused me to open the bin directory are really focus. In it, I had put several files for mbU and nH, but lo and behold, there was also, ProjectName.dll. It had also copied the referenced .dll files there as well. Lights went on.

Later, the nHibernate fiasco told me that I needed to have its XML file as an "embedded resource". I did that (to no avail) and realized that must mean that it can pack things into the .dll, as well as put them nearby as references. It also told me that I needed to build explicitly because VS wouldn't automatically detect changes to XML as it (sometimes) does to code.

Cool beans. I am now a reference expert.

Asking myself how mbUnit would know about my real ProjectName, I did two things: 1) added a reference to the ProjectName.dll, even though it was in some other project's directory, and 2) put a using statement that referred to the test target's namespace (using ProjectName.Models) and then instantiated a "TestTarget xxx=new TestTarget();" It had a method that was there for just this purpose and Voila!, I was able to test and will be able to test from now on.

This is killer.

Stuck.

Sending data from JMVC back to C# is a trivial exercise. I have, fortunately, been handicapped with sufficient approval that I feel like I am no longer experimenting. That means that I am less inclined to do things that are mere scaffolding as I have been doing so far. I was thrilled to merely be able to receive a data structure and then display it before. Now I want the next step to be one that contributes more fully to the development cycle.

My plan had been to make JMVC present an input box, collect the data and send it back to the database. I called it, "completing the loop." As I worked into that, I realized that I would need to implement a number of structures to do that and, vis paragraph one here, I felt like I would be wasting time to do that as a mere demonstration. After a day of huffing and puffing, I decided that I needed to decided if the target database was fronted by nHibernate. I also felt like the process of figuring that out would answer a lot of questions.

Then I found a great nHibernate tutorial. It's fourteen video lessons and, if the first is typical, it is an incredibly valuable contribution to the world. Not only did the guy give me view of nH, but also I got to witness someone actually using Visual Studio, see his code and hear how .NET types think about things. I finally understand .dll, for example, and how resources are connected and many other things.

Among those is a more practical understanding of unit testing. Instead of making a display page to show us that his demo worked, he fired up a unit test. This is a huge Eureka! moment for me. I've tried to understand Test Driven Design for a long time, but, having not really seen much in the way of unit testing, I couldn't quite understand how it works. Now I have a clue.

I also saw that this guy, for whom I am developing an intense, loving Stockholm Syndrome, uses mbUnit for his unit testing and made it look easy. I investigated. Most people think that Visual Studio's new, built-in, unit testing will, in the future, be a fully competent framework. A lot think that, even when that happens, the reliance on a single development team at Microsoft will prevent it from being as rich a tool as the other leading one, mbUnit. Since both my colleagues and I are all about open source, that die is cast.

I downloaded mbUnit and installed it. That was the last happy moment of my day.

I haven't yet been able to figure out how to get either VS unit testing or mbUnit to see my code. I spent hours flogging myself about it. I'd give it up but really think that it is going to be the best way to make it so that database server development can be truly separated from control/view development. I have always been confident that unit testing is a crucial technology. Now I think it will make my life easier and quicker. If only I could make it work.

It Gets Difficult

Going from LAMP to .NET is not the only big challenge here. No, I am on the big rollercoaster. Everything I've ever known is out the window. Everything I see is new. It is the most amazing thing. One thing is the use of frameworks. I've never, ever really done it before. Big change in mindset to open my head to other people's ideas with the knowledge that I am stuck with them for the foreseeable future. All this MVC? Probably not the biggest change, but still, I've taken it on at a level that I never considered before. ORM is going to kick my ass. I'm getting ready to be serious about it, but each time I dig, I find the layers of complication exploding and shrink from the immensity. I don't know if it's the right thing to do but I do know. It is. But I'm not sure because it looks so hard.

But this all adds up to a whole new consideration of architecture because, this time, I'm not making a website, I'm writing an application. There are a lot of increased responsibilities with that. Domain driven design. Test driven design. Unit testing. We haven't even talked about actually accomplishing the business needs.

We did a technical review of the project yesterday. I'm happy to say that it went well. I spent three hours telling the boss and the rest of the team why I have been doing all the things I've recorded in this blog. Some are not really web types and so I got to review my understanding of MVC. I talked about the alternatives I've rejected. Discussed the organization of my code, it's structures and my plan to move it forward. The rest of the folks were nice about it and I got a green light to move forward with this architecture.

I'm still really in R&D mode. If you look below, you'll see that I've basically implemented a category/detail demo. It uses .NET MVC to operate the database and JavascriptMVC for the user interface and control. This is strictly a business app and does not have to work for the random public. That means I can write the app in Javascript and use C# as a data service. This is very cool.
At present the app data flows one direction, outbound. My next step is to get some new data and send it back to the server then database. Time to close the loop.

I spent today putting my fingers to the keyboard, sighing. Pausing. Waiting. Getting a cup of coffee. Putting my fingers on the keyboard. Sighing. ... It turns out that there a lot of considerations for this next step. Not only have I been taken away from the safety of still just trying to figure out what to do and placed in the category of, We like it. How long will it take, but I now have to push through territory I already own.

On the outbound journey, I was pioneering. Everything that worked at all was a gift. Simple and barely working was all I want. Now, though, I have a lot of the structures in place and feel like I need to improve everything on the way. That is, it's nice to figure out how to send data back to the database but I don't really want to invest the time without also making progress toward the good solution.

Bottom line on this blab is that I spent all day to realize that my big issue is the one I have always known is the most important single one: data structures. I am stymied because I don't know what sort of objects I am going to pass between C# and Javascript. Objects, obviously. In JSON, of course, but what objects and where are they.

After a while, I started to experiment with the objects I am receiving from the database already. (I had, of course, done so before but really just grabbed the bits I needed to display). It turns out that doing a LINQ query for a set of categories also brings along a set of details, just as one would hope. I can get that and, I suppose, operate on it. That is, as the user enters data, I can store it in that object and eventually send it back to the server. I will have to write a method that plucks it apart and saves the stuff that's changed, make it persist.

Persist!!! That's what nHibernate does. I cycle around my nightmare and try to find a chink in nHibernate's armor. That puts me into despair and, when I'm in despair I procrastinate. In this case, I read a lot about nHibernate and it's handy band of added tools. But, it also caused me to pass through SOA (service oriented architecture) and SOAP. Suddenly I think, maybe I'm resisting the wrong thing. Maybe this should be a SOAP interface. Grab me a WSDL, instantiate a few objects, talk to the database via SOAP.

Is is that making a mountain out of a molehill? My head is going to explode.