Since last week: Color-picking, new templates, various improvements

Today, I tried writing an outline of this post in OmniOutliner 3.6 professional, with links to Trac Changesets in its "notes" -- unfortunately, it was difficult to export it to a format suitable for Blogger; the best I could do was export to RTF, then export that to HTML, and remove all the newlines with gvim; the result was still not worth posting.

Without further ado, here's the changes:

• Color-picking!
• While it's not in any of the useful templates yet, we have a working color-picker, based on LP Dalton's widget.
• New Templates
• Jon has been re-creating his textarea-base editor under the new template system; you can test-drive it here (though the link will eventually break when he picks a better name than "barebonesEdDrag").
• SVN Branching Scheme
• New development branches in SVN will be named after their creation date, folowed by a short description of what's new; the first such branch is called 20060627-templates.
• Numerous bug-fixes
• Clients are less likely to step on each others' toes, TinyMCE interferes less with dragdrop, and many other things are generally smoother than they used to be.
• It's easier to make new functions
• With the addition of utlility-functions like "get_element_from_argument" and "find_twoway_parent", it's easy to make robust UI functions.
• A combination "get_twoway_element_from_argument()" is on the way.
• TwoWay elements are now marked with the "twoway" attribute, such as <div id="'A'" twoway="'watched'"></div>. This makes it easier to know which elements are important, and is a good platform for other twoway-related metadata.
• To ask the server to keep track of an element, all you need to do is make sure that element has an ID, and pass it to watch().
• In the future, watch() will be able to attach an ID to elements that don't have one.
• Documentation!
• We've been working on a Tutorial section on TwoWayWeb, but it's not on a public server yet; this will likely lead to the ability to copy pages from one server to another with minimal fuss.
• I think it would be really neat to add a few lines to a TiddlyWiki and make it TwoWay-capable. More on that idea later.
  
• Selenium-based testing is on the way
• I've added Selenium and a couple basic tests to see hw easily I can drive our application; more will follow, I hope.
• The main problem with Selenium right now is that it can't easily deal with TinyMCE, Drag-dropping or precise clicking. Maybe someone will fix this while we wait; maybe I'll fix it in my off-time. However, it's great for most other UI features.

The plan from here: TicTacToe, as simple as possible.

Next, I am going to create a sample TicTacToe template to demonstrate how easily one can develop applications over TwoWayWeb.

It should be this easy:

• Make a new, empty template. Call it tictactoe.html.
  
• Include the 4 necesary Javascript files. (We plan to bring this down to 1).
  
• Create a 3x3 table, with some ID ("TicTacToeBoard").
• In each cell, make a DIV with the attribute twoway="true".
• For now, make their styles all "background: yellow; width: 100em; height: 100em;"
• Alternately:
• Define a create_first_element function, to create one DIV in each cell of the table.
• Tell the server about each one by calling "TwoWay.watch( element );"
  
• Load a page with that template, and marvel at your 3x3 grid of boxes.
• Open that page in another browser, and change the color of one box with FireBug or something. Watch the change appear on your other browser.
  
• Add a method that cycles an element's innerHTML between X, O and [blank].
• Attach that method to the ONCLICK of each DIV.
• Point a friend at some page with your new template, and play!
  

Other plans:

• Reduce the 4-script include in barebones to a single script.
• Make debug() create its own debugging DIV if necessary.
• Make a logging function, and have it do the same.
• Update all instances of "if (LOGGING)".
• Add a --onescript option to Launch2way.py, to match the new feature in 2way
• Possibly add a main() to 2way.py that the launcher can call with profiling, rather than invoking a new Python process.
• Move most of the Python sources to py/ or something, to clean up the directory.
• Make a short note that explains what each file does.
• Let each page "know" about a preferred viewing template.
• Refactor saver.py (waaay down there on the list).
• Look into using SvnX or something for easier SVN access on OS X.
  

Changes since Tuesday: UI Split, Templates, Bug-fixes

Wow, so much has happened since Tuesday's post. This entry is current for revision #577.

Since there have been so many changes, I'm going to describe the next steps in my next blog post.

• First, you can check on the (usually) latest experimental version of 2way at http://twowayweb.rit.edu:8081/
• As of rev. 558, Python sources download with a MIME-type of text/plain.
• All the silly <pre> tags have been removed from the Python sources.
  
• In revision 559, we moved several import functions from 2wayUI.js to 2wayCore.js, preparing the way for drop-in alternate interfaces.
• Rev. 561 was a continuation of this; it also established my debug() as the canonical dump-information-to-screen function.
  
• We then committed an OmniGraffle file with our documentation, showing the approximate coupling between each function.
• Changesets 562 through 568 added a basic template system and fixed some related bugs.
• The Barebones template was added in rev. 569, and modified several times after.
  
• You can now select the Barebones template by appending "?template=barebones.html" to any TwoWayWeb URL.
• The server now sends entire pages to clients with new (random) IDs (as it was meant to long ago).
• bootStrap's power has been greatly reduced, and it no longer takes a 'payload' parameter (though it can still use the 'payload' global variable, if it is declared by the template).
• My initial "if (function_name)" callbacks were wrong -- one can only check for properties on an object like that. Relevant sections now use try-blocks instead (r571, 572).
• create_first_element now happens in tick() instead of bootStrap(), though it doesn't do anything if "deleted" elements are merely hidden.
• Finally, new pages are loaded with the correct template, and the 2wayDaemon only creates the "first" element if it has already learned from the server that the page is empty.
  

TinyMCE rolled in!

Yesterday, I finally finished rolling TinyMCE support into the main trunk of development - check it out!

To edit a panel, double-click on it; TinyMCE will appear in a new panel, right over the one you clicked on. to stop editing, double-click anywhere outside the editor.

There are still a few minor problems with this setup:

• Double-clicking elsewhere (like selecting a word from the toolbar's Javascript interpreter) can accidentally close the editor.
• Short of choosing a new way to close editors, I don't know if there's a way around this.
  
• Drag-drop is a little flaky, and we're not sure why.
• It would be nice to monitor all Javascript events somehow, so we could figure out what's going on.
• Text in the editor is significantly smaller than text outside.
• This is annoying enough to warrant a fix, but we haven't looked into how yet.
• We can't use [[brackets]] to make iframes anymore.
• An enhancement for that is on the backburner. Let me know if it should go to the frontburner.

Other news:

• allscripts.js is now optional, and disabled by default.
• It should be enabled (by default?) for production servers, but it's a lot easier to debug without it -- which brings us to...
  
• One of the first changes I ever made to TwoWayWeb is the addition of Launch2way.py.
• This script handles command-line arguments for 2way.py; currently, the only argument it takes is -p, which enables profiling for 2way and writes the results to a file in the stats/ directory.
• As of revision 523, Launch2way.py unpacks CherryPy 2.2 from the vendor/ directory and installs it to "cherrypy", so 2way.py can access CherryPy without affecting the rest of a server.
• It does this by executing freeze_cherrypy.sh.
  
• I would like this or something similar to become the preferred starting-point for 2way.py, so we can deal with command-line arguments and one-time setup apart from the server iteslf.
• As I hinted above, there should also be a switch to enable or disable the allscripts.js "Glob o' JavaScript".
  
• Trac is choking on some revision links. I think this is because I told it our repository is "svn-repository/2way", when I really just want it to ignore the other top-level directories in subversion. I'm not sure how to best fix this.
• We've started organizing our repository more: after breaking the main branch once, we started using other branches for experiments.
• I've started graphing the dependencies between each function in our Javascripts. More on that later.
• Finally, we want to prevent visits to nonexistant pages from creating the pages.
• Wikis generally say "This doesn't exist yet. do you want to create it?"
• If we went that route, we could also suggest probable typos.
  
• Apparently, earlier versions didn't store anything in the database unless a page was changed.

(This blogpost applies to revision #555.)

WYSIWYG Editing en-route

Over the past few days, I've been progressing towards WYSIWYG editing with TinyMCE.

Since my last post, I have:

• Added a raw-HTML editing button to TinyMCE (via its 'HTML' button)
• Completed more pre-integration demos: 2wayTinyMCE.html, dragdrop-tinyMCE.html and tiny_mce_test.html.
• TinyMCE now starts on doubleclick, and stops when the use doubleclicks anywhere else. To keep these events from stepping on each other's toes, the start-handler is built to kill any event handed to it.
• For a more robust solution, I could probably bind the close-event only when there is an editor open, and remove that binding when it is activated. That would end the need for an event-parameter.
  
• As of revision 535, the CloneMe! button duplicates the original element's contents.
• As of revision 536, this is no longer necessary for TinyMCE.
• In Revision 536, the editor made its own wrapper-DIV to start in, so modifications to the page structure are no longer necessary.
• Editable elements are now marked by setting their "editable" property (to anything but the empty string, "").
• In revision 537 of EditMe.js.html, I changed the demo to (mostly) work by giving ondblclick() an event instead of the editable panel's ID.
• Finally, I have some uncommitted changes that attempt to roll the editor into TwoWayWeb proper.
• Unfortunately, since the editor is inserting itself into a monitored DIV, the TinyMCE interface is visible (though mangled) on other browsers. To combat this, the editor will once again become a floating window.
  

From here, I shall:

• Change my "document.onload =" to "document.addEventListener('load', .... )"
• Create "make_popup_editor()" to start TinyMCE in a top-level (dragdroppable) DIV, with an initial position and size equal to the edited node.
• After the demos work, see if that change is enough to make TinyMCE work smoothly on 2way.

Along the way, FireBug 0.4 has been invaluable. With the advent of its debugger, I can step through javascript code with all local variables displayed, and easily inspect others with its console.

I have also learned a lot about the Javascript event model, particularly with the help of this Quirksmode.org page. It's a good read for anyone who wants to handle events, particularly if more than one element is listening for the same event.

Week 2: Configuring OS X

Today, I finished setting up our Trac server and configured it to run on boot. There were a few Gotchas:

• I can't run Trac through mod_python, because of some strange OS X bug. It's a known issue, and I left a comment on Trac bug #2629 to let them know that it's still broken with recent versions of PySqlite.
• I tried to get Apache to rewrite all requests on port 80 to our Trac server - intending to "mount" tracd on /trac, and redirect / to /trac until we have something better - but it didn't work. Here is my current attempt:
  
  RewriteEngine on
  RewriteRule ^/(.*) http://twowayweb.rit.edu:9000/twowayweb/$1 [R]
  
  
• The first time I loaded it, I forgot to use "launchctl load -w" to make sure it started at the next reboot - oops. It's working now.
  
• Drat, I just noticed that Login is broken. I'll fix it tomorrow.
  

Subversion and Apache are also loading on boot, but twowayweb isn't; it probably still deadlocks under heavy load, so I don't want to run it as a system process until we can reliably restart it automatically.

For that matter, I want to simulate high load on demand, without getting 30 people in one place to move windows around willy-nilly. Unfortunately, I don't think traditional tools like siege would apply, because as far as I can tell, the problem is triggered by too much javascript action.

A browser-driver like Selenium might work, but I'd rather use something I can drive from the command-line... I remember reading about a no-output X server that could be used to launch Firefox (or whatever) sessions without cluttering the display. That might be just what I need, along with an automatic driver.

The other problem with Selenium is that it's missing a few events, most notably doubleclick and drag. I'm sure we could add them if it's worth it, but it might not be.

In short, I want a javascript-capable program-driven web browser.

In other news, I'm almost done with a drop-in WYSIWYG replacement for the current text editor. As far as I can tell, there are only a few issues remaining:

• Since the editor overlays the current panel rather than making another floating one, we need a new way to close editors.
• Several of our javascripts redefine the $() function. I want to get DRY again as soon as possible.
• When open in TinyMCE, text appears much smaller than when the editor is closed:
  
  
• TwoWayweb's "clone" function currently obliterates the innerHTML of its target element, but my integraton of TinyMCE requires another nested DIV around a panel's contents. Clone needs to be fixed to reflect this.
• Finally, people might want to edit raw HTML from TinyMCE -- I should re-enable the 'code' button.

That's it for now; i'm going home. I might get my Macbook in time to bring it in tomorrow, but I suspect I'll have to wait until Thursday.

Week 1 wrap-up

Today, I started to get our old host back to normal after an OS reinstallation:

• Installed XCode 2.3, from the Apple Developer Site
  
• Installed DarwinPorts
• Installed Python24, screen and coreutils (I much prefer GNU ls over the BSD version).
• Installed Subversion 1.3.2 (with Martin Ott's OS X installer), SubEthaEdit (pus the command-line tool) and Bwana.
• Insalled Camino and Firefox.
• Firefox extensions: Web Developer Toolbar and FireBug.

I also have TinyMCE working in a sandbox; since we decided that we don't really need concurrent editors, I don't need to worry about the associated bugs. Instead, I will look into creating a new TinyMCE theme, which attaches the tool-bar to the top of the panel being edited. The editor would work in-place, and only run on a single panel at once.

I actually have some misgivings about that:

• What about when someone else moves a panel? We don't want the editor to go flying across the screen unexpectedly.
• Right now, if two people edit the same panel, one of them is wasting his time. It would be really nice if we attempted to merge changes that don't affect the same paragraph.

I also submitted the co-op paperwork today, and should be on a regular 10-6 schedule starting Monday. In my off-moments of work, this weekend and beyond, I will clean up the Javascript code (with yesterday's Lint tools), run PyLint on the Python sources and continue to forgure out how it all works together.

Bugfixes and Cleanup

This morning, I startd really using my local SVN repository again, pending SourceForge approval; not doing so yesterday turned out to be a significant performance hit; it definitely didn't help that the AC adapter for my laptop died the day before.

I immediately rolled in the bug-fixes of the past two days, plus a few more:

• most notably, 2wayDaemon's polling no longer spams the DOM with empty elements.
  
• 2WayUI's o2e() now inserts elements with the correct tag name.
• The loopback-link creation is both simpler and more reliable.
• Several redundant and/or outdates files are gone.
• The /params service is functional once again.
• Exernal dependencies (currently CherryPy and TinyMCE) have tarballs in /vendor.

I'm currently wrapping my head around the UI calls so I can get TinyMCE support working: first I'll get one editor to show up and show changes near-instantly, then I'll expand the function to allow any number of concurrent editors. If I recall correctly, the second part may be bumpier than one would think, but I'm less worried about it with better debugging tools than I had the last time I tried this -- most notably FireBug 0.4, with stack traces for errors.

I also found a few Javascript Lint tools yesterday:

• Javascript Lint is pretty, but lax.
• JSLint is ugly, but far more useful.
• This Blogpost mentions a few other tools I should investigate.
  

Things to do until approved by SourceForge

• Now:
  
• Integrate TinyMCE as a WYSIWYG editor with near-continuous updates.
• Evaluate Selenium (perhaps driven by Python) as a testing framework.
• Later:
  
• Other Panel UI improvements
• Color-picking
• Snap-on-drag (by 5 or 10 pixels)
• Resizing-handles, also snapping
• Display handles and menu-widget on mouseover
• Understand how the network arcchitecture works
• Evaluate Pythonic database drivers (preferably with sqlite support)?
  

Speaking of the network architecture, it has some silly behavior: when the user loads a new page, the Javascript polling system fetches the page's contents -- they don't come with the original page, so people need to wait longer to read anything and users with broken AJAX can't even see what they can't edit.

So I want to move the original DOM assembling to the webserver, and examine how data is moving around to discover other silly behaviors.

I'm also going to examine the Javascript libraries currently in use -- some of the nice bits of my HCI2 project came for free with Scriptaculous (which depends on Prototype)

Update:

Spotty WiFi reception has prompted me to investigate distribution methods early, so here are some relevant links:

• Python's own Distutils
• py2exe will bundle a Python script and any required libraries into a single EXE file -- hopefully it runs faster than rubyscript2exe; if so, it could be immediately useful for times when I just want to test things without modifying the source.
• A 4-year-old email on python-list, discussing the above.

I just noticed something odd with Firefox's DOM Inspector -- the Panels' node types are "UNDEFINED", and more are being added to the DOM constantly. Whatever is going on, it should probably be fixed.

From here, I want to:

• Determine a target application for TwoWayWeb,

• Improve TwoWayWeb's current interface in light of that application,

• Improve performance for said application (for at least 100 concurrent users, possibly with a new architecture),

• and prepare the software for distribution, such that we can track how it disseminates and alert users to upgrades.

Since I'm working on TwoWayWeb with Jon Schull this summer, I should have a blog dedicated to that work - and here we are.

I've started by requesting a SourceForge project, and am familiarizing myself with the source and other relevant documents:

• US Patent #6,266,654, "Method for tracking software lineage"
• His Macroscope Manifesto
• "Seeing and Tuning Social Networks," which pertains to the above Manifesto
• "Predicting the evolution of digital rights, digital objects, and digital rights management languages"
• "Multichat: persistent, text-as-you-type messaging in a web browser for fluid multi-person interaction and collaboration."
• "Finally the TwoWayWeb: a straw man roadmap."

Next, I'm going to familiarize myself with the twowayweb sources and consider how easy i will be to integrate the results of my Interactoin Design project, most notably a TinyMCE for WYSIWYG editing and a better color-picker.