Changes since last Thursday

This covers revisions 635 - 701.

Various Improvements to templates:

• ltwt.html - copied from barebonesEdDrag.html, ** later diverged **
• TicTacToe, tictac2 -- TicTac2 demonstrates a problem with o2e()
• tic.html -- super-useful debug template! Click to change element, LOGGING enabled by default.
  

Core Javascripts:

• Simplified clone(), edit(), hide()...
• ...touch(), untouch()
• Added get_twoway_element_from_argument();
• Better error-announcing from template-hooks
• Auto-discovery of elements with TwoWay attribute set
• watch()ing an element without an ID will auto-generate an ID for it.
• Replaced subO() with a non-broken, cleaner get_filtered_object()
• Propagating onclick(), ondblclick() event handlers with pack_eventS(), unpack_events()
• LOGGING is more verbose (myRandomNumber, current time)
• Code-cleanup, particularly for huge functions
• TODO: Remove vestigial payload=$payload from templates, remove payload-processing from bootStrap(), veryify no-breakage
  

Template-specific Javascripts:

• 2wayUI: Snapping to grid, by 5 pixels

Python stuff:

• We split up 2way.py's default() into smaller functions
• Minor code-cleanup -- comments, spacing and the like.
• Saving snapshots of pages as templates(!!!)
• Template-specific panels are stored in templates/templateName.data
• When a page is first saved, we copy all elements form the template to the new page.
• interp() is more robust and doctested. - we don't need to use $$ for non-variables anymore.
• saver is more robust -- returns '{}' when we ask for a nonexistant file.

In the works:

• Some attempts at a rigt-click menu
• STILL cleaning up inconsistent newlines; SubEthaEdit 2.5 is very helpful.

Selenium:

• Added doubleclick support (TODO: Note where I found that code!)

Also:

• We realized that referring to elements by ID is much more stable than storing references to Element objects (they can get stale after ticks)

Since last week: Color-picking, new templates, various improvements

Today, I tried writing an outline of this post in OmniOutliner 3.6 professional, with links to Trac Changesets in its "notes" -- unfortunately, it was difficult to export it to a format suitable for Blogger; the best I could do was export to RTF, then export that to HTML, and remove all the newlines with gvim; the result was still not worth posting.

Without further ado, here's the changes:

• Color-picking!
• While it's not in any of the useful templates yet, we have a working color-picker, based on LP Dalton's widget.
• New Templates
• Jon has been re-creating his textarea-base editor under the new template system; you can test-drive it here (though the link will eventually break when he picks a better name than "barebonesEdDrag").
• SVN Branching Scheme
• New development branches in SVN will be named after their creation date, folowed by a short description of what's new; the first such branch is called 20060627-templates.
• Numerous bug-fixes
• Clients are less likely to step on each others' toes, TinyMCE interferes less with dragdrop, and many other things are generally smoother than they used to be.
• It's easier to make new functions
• With the addition of utlility-functions like "get_element_from_argument" and "find_twoway_parent", it's easy to make robust UI functions.
• A combination "get_twoway_element_from_argument()" is on the way.
• TwoWay elements are now marked with the "twoway" attribute, such as <div id="'A'" twoway="'watched'"></div>. This makes it easier to know which elements are important, and is a good platform for other twoway-related metadata.
• To ask the server to keep track of an element, all you need to do is make sure that element has an ID, and pass it to watch().
• In the future, watch() will be able to attach an ID to elements that don't have one.
• Documentation!
• We've been working on a Tutorial section on TwoWayWeb, but it's not on a public server yet; this will likely lead to the ability to copy pages from one server to another with minimal fuss.
• I think it would be really neat to add a few lines to a TiddlyWiki and make it TwoWay-capable. More on that idea later.
  
• Selenium-based testing is on the way
• I've added Selenium and a couple basic tests to see hw easily I can drive our application; more will follow, I hope.
• The main problem with Selenium right now is that it can't easily deal with TinyMCE, Drag-dropping or precise clicking. Maybe someone will fix this while we wait; maybe I'll fix it in my off-time. However, it's great for most other UI features.

The plan from here: TicTacToe, as simple as possible.

Next, I am going to create a sample TicTacToe template to demonstrate how easily one can develop applications over TwoWayWeb.

It should be this easy:

• Make a new, empty template. Call it tictactoe.html.
  
• Include the 4 necesary Javascript files. (We plan to bring this down to 1).
  
• Create a 3x3 table, with some ID ("TicTacToeBoard").
• In each cell, make a DIV with the attribute twoway="true".
• For now, make their styles all "background: yellow; width: 100em; height: 100em;"
• Alternately:
• Define a create_first_element function, to create one DIV in each cell of the table.
• Tell the server about each one by calling "TwoWay.watch( element );"
  
• Load a page with that template, and marvel at your 3x3 grid of boxes.
• Open that page in another browser, and change the color of one box with FireBug or something. Watch the change appear on your other browser.
  
• Add a method that cycles an element's innerHTML between X, O and [blank].
• Attach that method to the ONCLICK of each DIV.
• Point a friend at some page with your new template, and play!
  

Other plans:

• Reduce the 4-script include in barebones to a single script.
• Make debug() create its own debugging DIV if necessary.
• Make a logging function, and have it do the same.
• Update all instances of "if (LOGGING)".
• Add a --onescript option to Launch2way.py, to match the new feature in 2way
• Possibly add a main() to 2way.py that the launcher can call with profiling, rather than invoking a new Python process.
• Move most of the Python sources to py/ or something, to clean up the directory.
• Make a short note that explains what each file does.
• Let each page "know" about a preferred viewing template.
• Refactor saver.py (waaay down there on the list).
• Look into using SvnX or something for easier SVN access on OS X.
  

Changes since Tuesday: UI Split, Templates, Bug-fixes

Wow, so much has happened since Tuesday's post. This entry is current for revision #577.

Since there have been so many changes, I'm going to describe the next steps in my next blog post.

• First, you can check on the (usually) latest experimental version of 2way at http://twowayweb.rit.edu:8081/
• As of rev. 558, Python sources download with a MIME-type of text/plain.
• All the silly <pre> tags have been removed from the Python sources.
  
• In revision 559, we moved several import functions from 2wayUI.js to 2wayCore.js, preparing the way for drop-in alternate interfaces.
• Rev. 561 was a continuation of this; it also established my debug() as the canonical dump-information-to-screen function.
  
• We then committed an OmniGraffle file with our documentation, showing the approximate coupling between each function.
• Changesets 562 through 568 added a basic template system and fixed some related bugs.
• The Barebones template was added in rev. 569, and modified several times after.
  
• You can now select the Barebones template by appending "?template=barebones.html" to any TwoWayWeb URL.
• The server now sends entire pages to clients with new (random) IDs (as it was meant to long ago).
• bootStrap's power has been greatly reduced, and it no longer takes a 'payload' parameter (though it can still use the 'payload' global variable, if it is declared by the template).
• My initial "if (function_name)" callbacks were wrong -- one can only check for properties on an object like that. Relevant sections now use try-blocks instead (r571, 572).
• create_first_element now happens in tick() instead of bootStrap(), though it doesn't do anything if "deleted" elements are merely hidden.
• Finally, new pages are loaded with the correct template, and the 2wayDaemon only creates the "first" element if it has already learned from the server that the page is empty.
  

TinyMCE rolled in!

Yesterday, I finally finished rolling TinyMCE support into the main trunk of development - check it out!

To edit a panel, double-click on it; TinyMCE will appear in a new panel, right over the one you clicked on. to stop editing, double-click anywhere outside the editor.

There are still a few minor problems with this setup:

• Double-clicking elsewhere (like selecting a word from the toolbar's Javascript interpreter) can accidentally close the editor.
• Short of choosing a new way to close editors, I don't know if there's a way around this.
  
• Drag-drop is a little flaky, and we're not sure why.
• It would be nice to monitor all Javascript events somehow, so we could figure out what's going on.
• Text in the editor is significantly smaller than text outside.
• This is annoying enough to warrant a fix, but we haven't looked into how yet.
• We can't use [[brackets]] to make iframes anymore.
• An enhancement for that is on the backburner. Let me know if it should go to the frontburner.

Other news:

• allscripts.js is now optional, and disabled by default.
• It should be enabled (by default?) for production servers, but it's a lot easier to debug without it -- which brings us to...
  
• One of the first changes I ever made to TwoWayWeb is the addition of Launch2way.py.
• This script handles command-line arguments for 2way.py; currently, the only argument it takes is -p, which enables profiling for 2way and writes the results to a file in the stats/ directory.
• As of revision 523, Launch2way.py unpacks CherryPy 2.2 from the vendor/ directory and installs it to "cherrypy", so 2way.py can access CherryPy without affecting the rest of a server.
• It does this by executing freeze_cherrypy.sh.
  
• I would like this or something similar to become the preferred starting-point for 2way.py, so we can deal with command-line arguments and one-time setup apart from the server iteslf.
• As I hinted above, there should also be a switch to enable or disable the allscripts.js "Glob o' JavaScript".
  
• Trac is choking on some revision links. I think this is because I told it our repository is "svn-repository/2way", when I really just want it to ignore the other top-level directories in subversion. I'm not sure how to best fix this.
• We've started organizing our repository more: after breaking the main branch once, we started using other branches for experiments.
• I've started graphing the dependencies between each function in our Javascripts. More on that later.
• Finally, we want to prevent visits to nonexistant pages from creating the pages.
• Wikis generally say "This doesn't exist yet. do you want to create it?"
• If we went that route, we could also suggest probable typos.
  
• Apparently, earlier versions didn't store anything in the database unless a page was changed.

(This blogpost applies to revision #555.)

WYSIWYG Editing en-route

Over the past few days, I've been progressing towards WYSIWYG editing with TinyMCE.

Since my last post, I have:

• Added a raw-HTML editing button to TinyMCE (via its 'HTML' button)
• Completed more pre-integration demos: 2wayTinyMCE.html, dragdrop-tinyMCE.html and tiny_mce_test.html.
• TinyMCE now starts on doubleclick, and stops when the use doubleclicks anywhere else. To keep these events from stepping on each other's toes, the start-handler is built to kill any event handed to it.
• For a more robust solution, I could probably bind the close-event only when there is an editor open, and remove that binding when it is activated. That would end the need for an event-parameter.
  
• As of revision 535, the CloneMe! button duplicates the original element's contents.
• As of revision 536, this is no longer necessary for TinyMCE.
• In Revision 536, the editor made its own wrapper-DIV to start in, so modifications to the page structure are no longer necessary.
• Editable elements are now marked by setting their "editable" property (to anything but the empty string, "").
• In revision 537 of EditMe.js.html, I changed the demo to (mostly) work by giving ondblclick() an event instead of the editable panel's ID.
• Finally, I have some uncommitted changes that attempt to roll the editor into TwoWayWeb proper.
• Unfortunately, since the editor is inserting itself into a monitored DIV, the TinyMCE interface is visible (though mangled) on other browsers. To combat this, the editor will once again become a floating window.
  

From here, I shall:

• Change my "document.onload =" to "document.addEventListener('load', .... )"
• Create "make_popup_editor()" to start TinyMCE in a top-level (dragdroppable) DIV, with an initial position and size equal to the edited node.
• After the demos work, see if that change is enough to make TinyMCE work smoothly on 2way.

Along the way, FireBug 0.4 has been invaluable. With the advent of its debugger, I can step through javascript code with all local variables displayed, and easily inspect others with its console.

I have also learned a lot about the Javascript event model, particularly with the help of this Quirksmode.org page. It's a good read for anyone who wants to handle events, particularly if more than one element is listening for the same event.

Week 2: Configuring OS X

Today, I finished setting up our Trac server and configured it to run on boot. There were a few Gotchas:

• I can't run Trac through mod_python, because of some strange OS X bug. It's a known issue, and I left a comment on Trac bug #2629 to let them know that it's still broken with recent versions of PySqlite.
• I tried to get Apache to rewrite all requests on port 80 to our Trac server - intending to "mount" tracd on /trac, and redirect / to /trac until we have something better - but it didn't work. Here is my current attempt:
  
  RewriteEngine on
  RewriteRule ^/(.*) http://twowayweb.rit.edu:9000/twowayweb/$1 [R]
  
  
• The first time I loaded it, I forgot to use "launchctl load -w" to make sure it started at the next reboot - oops. It's working now.
  
• Drat, I just noticed that Login is broken. I'll fix it tomorrow.
  

Subversion and Apache are also loading on boot, but twowayweb isn't; it probably still deadlocks under heavy load, so I don't want to run it as a system process until we can reliably restart it automatically.

For that matter, I want to simulate high load on demand, without getting 30 people in one place to move windows around willy-nilly. Unfortunately, I don't think traditional tools like siege would apply, because as far as I can tell, the problem is triggered by too much javascript action.

A browser-driver like Selenium might work, but I'd rather use something I can drive from the command-line... I remember reading about a no-output X server that could be used to launch Firefox (or whatever) sessions without cluttering the display. That might be just what I need, along with an automatic driver.

The other problem with Selenium is that it's missing a few events, most notably doubleclick and drag. I'm sure we could add them if it's worth it, but it might not be.

In short, I want a javascript-capable program-driven web browser.

In other news, I'm almost done with a drop-in WYSIWYG replacement for the current text editor. As far as I can tell, there are only a few issues remaining:

• Since the editor overlays the current panel rather than making another floating one, we need a new way to close editors.
• Several of our javascripts redefine the $() function. I want to get DRY again as soon as possible.
• When open in TinyMCE, text appears much smaller than when the editor is closed:
  
  
• TwoWayweb's "clone" function currently obliterates the innerHTML of its target element, but my integraton of TinyMCE requires another nested DIV around a panel's contents. Clone needs to be fixed to reflect this.
• Finally, people might want to edit raw HTML from TinyMCE -- I should re-enable the 'code' button.

That's it for now; i'm going home. I might get my Macbook in time to bring it in tomorrow, but I suspect I'll have to wait until Thursday.

Week 1 wrap-up

Today, I started to get our old host back to normal after an OS reinstallation:

• Installed XCode 2.3, from the Apple Developer Site
  
• Installed DarwinPorts
• Installed Python24, screen and coreutils (I much prefer GNU ls over the BSD version).
• Installed Subversion 1.3.2 (with Martin Ott's OS X installer), SubEthaEdit (pus the command-line tool) and Bwana.
• Insalled Camino and Firefox.
• Firefox extensions: Web Developer Toolbar and FireBug.

I also have TinyMCE working in a sandbox; since we decided that we don't really need concurrent editors, I don't need to worry about the associated bugs. Instead, I will look into creating a new TinyMCE theme, which attaches the tool-bar to the top of the panel being edited. The editor would work in-place, and only run on a single panel at once.

I actually have some misgivings about that:

• What about when someone else moves a panel? We don't want the editor to go flying across the screen unexpectedly.
• Right now, if two people edit the same panel, one of them is wasting his time. It would be really nice if we attempted to merge changes that don't affect the same paragraph.

I also submitted the co-op paperwork today, and should be on a regular 10-6 schedule starting Monday. In my off-moments of work, this weekend and beyond, I will clean up the Javascript code (with yesterday's Lint tools), run PyLint on the Python sources and continue to forgure out how it all works together.