mykzilla

URL Has Been Changed

2016-01-11T08:57:00.001-08:00

The URL you have reached, http://mykzilla.blogspot.com/, has been changed. The new URL is https://mykzilla.org/. Please make a note of it.

Introducing PluotSorbet

2015-06-23T16:05:00.000-07:00

PluotSorbet is a J2ME-compatible virtual machine written in JavaScript. Its goal is to enable users you run J2ME apps (i.e. MIDlets) in web apps without a native plugin. It does this by interpreting Java bytecode and compiling it to JavaScript code. It also provides a virtual filesystem (via IndexedDB), network sockets (through the TCPSocket API), and other common J2ME APIs, like Contacts.

The project reuses as much existing code as possible, to minimize its surface area and maximize its compatibility with other J2ME implementations. It incorporates the PhoneME reference implementation, numerous tests from Mauve, and a variety of JavaScript libraries (including jsbn, Forge, and FileSaver.js). The virtual machine is originally based on node-jvm.

PluotSorbet makes it possible to bring J2ME apps to Firefox OS. J2ME may be a moribund platform, but it still has non-negligible market share, not to mention a number of useful apps. So it retains residual value, which PluotSorbet can extend to Firefox OS devices.

PluotSorbet is also still under development, with a variety of issues to address. To learn more about PluotSorbet, check out its README, clone its Git repository, peruse its issue tracker, and say hello to its developers in irc.mozilla.org#pluotsorbet!

simplify asynchronous method declarations with Task.async()

2014-03-28T16:58:00.002-07:00

In Mozilla code, Task.spawn() is becoming a common way to implement asynchronous operations, especially methods like the greet method in this greeter object:

let greeter = {
  message: "Hello, NAME!",
  greet: function(name) {
    return Task.spawn((function*() {
      return yield sendGreeting(this.message.replace(/NAME/, name));
    }).bind(this);
  })
};

Task.spawn() makes the operation logic simple, but the wrapper function and bind() call required to start the task on method invocation and bind its this reference make the overall implementation complex.

Enter Task.async().

Like Task.spawn(), it creates a task, but it doesn't immediately start it. Instead, it returns an "async function" whose invocation starts the task, and the async function binds the task to its own this reference at invocation time. That makes it simpler to declare the method:

let greeter = {
  message: "Hello, NAME!",
  greet: Task.async(function*(name) {
    return yield sendGreeting(this.message.replace(/NAME/, name));
  })
};

With identical semantics:

greeter.greet("Mitchell").then((reply) => { ... }); // behaves the same

(And it avoids a couple anti-patterns in the process.)

Task.async() is inspired by ECMAScript's Async Functions strawman proposal and C#'s Async modifier and was implemented in bug 966182. It isn't limited to use in method declarations, although it's particularly helpful for them.

Use it to implement your next asynchronous operation!

qualifications for leadership

2014-03-27T13:14:00.000-07:00

I've been surprised by the negative reaction to Brendan's promotion by some of my fellow supporters of marriage equality. Perhaps I take it too much for granted that Mozillians recognize the diversity of their community in every possible respect, including politically and religiously, and that the only thing we share in common is our commitment to Mozilla's mission and the principles for participation.

Those principles are reflected in our Community Participation Agreement, to which Brendan has always shown fealty (since long before it was formalized, in my 15-year experience with him), and which could not possibly be clearer about the welcoming nature of Mozilla to all constructive contributors.

I know that marriage equality has been a long, difficult, and painful battle, the kind that rubs nerves raw and makes it challenging to show any charity to its opponents. But they aren't all bigots, and I take Brendan at his word and deed that he's as committed as I am to the community's inclusive ideals (and the organization's employment policies).

As Andrew Sullivan eloquently states in his recent blog post on Religious Belief and Bigotry:

"Twenty years ago, I was confidently told by my leftist gay friends that Americans were all anti-gay bigots and would never, ever back marriage rights so I should stop trying to reason them out of their opposition. My friends were wrong. Americans are not all bigots. Not even close. They can be persuaded rather than attacked. And if we behave magnanimously and give maximal space for those who sincerely oppose us, then eventual persuasion will be more likely. And our victory more moral and more enduring."

I'm chastened to admit that I substantially shared his friends' opinion twenty years ago. But I'm happy to realize I was wrong. And perhaps Brendan will one day do the same. Either way, he qualifies to be a leader at any level in the Mozilla community (and organization), as do the many other Mozilla leaders whose beliefs undoubtedly differ sharply from my own.

from Webapp SDK to r2d2b2g, Firefox OS Simulator, and the App Manager

2013-10-15T17:05:00.003-07:00

A little over a year ago, on August 31, 2012, I brainstormed the outline of a "Webapp SDK":

That outline was the genesis for the r2d2b2g experiment, which built the Firefox OS Simulator, whose initial version hit the web on September 14, 2012 and which has gone through numerous iterations since then as we evaluated various features to enhance app development.

And that experiment spurred Mozilla's Developer Tools group, particularly its nascent App Tools team, to build Firefox's new App Manager, which landed last month and was introduced today on Hacks!

Despite the twisty passage from experiment to product, that initial outline bears a surprising resemblance to the App Manager feature set. The Manager checks off three of the four features on the outline's primary list—"start Gaia in B2G," "package app," and "test app in Gaia/B2G"—plus a few on its secondary list, like "debug from Firefox" and "test on mobile device," with "edit manifest in GUI" well underway over in bug 912912. And the Simulator continues to provide B2G/Gaia via an easy-to-install addon that integrates with the Manager.

Like any good product of a successful experiment, however, the Manager's reach has exceeded its progenitor's grasp! So it also gives you access to pre-installed apps, lets you take screenshots of device/Simulator screens, and will doubtless continue to sprout handy features to make app development great.

So kudos to the folks who built it, and long live the App Manager!

fixing this morning's mach OS X psutil bustage

2013-08-23T10:55:00.001-07:00

If mach is broken in your mozilla-central clone on Mac OS X this morning:

08-23 10:20 > ./mach build
Error running mach:

    ['build']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.

You should consider filing a bug for this issue.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

AttributeError: 'module' object has no attribute 'TCPS_ESTABLISHED'

File "/Users/myk/Mozilla/central/python/mozbuild/mozbuild/mach_commands.py", line 293, in build
    from mozbuild.controller.building import BuildMonitor
File "/Users/myk/Mozilla/central/python/mozbuild/mozbuild/controller/building.py", line 22, in <module>
    import psutil
File "/Users/myk/Mozilla/central/python/psutil/psutil/__init__.py", line 95, in <module>
    import psutil._psosx as _psplatform
File "/Users/myk/Mozilla/central/python/psutil/psutil/_psosx.py", line 48, in <module>
    _TCP_STATES_TABLE = {_psutil_osx.TCPS_ESTABLISHED : CONN_ESTABLISHED,

Then you've been bit by the fix for bug 908296. To resolve the bustage, run this command in your Hg clone:

hg status -in python/psutil | xargs rm

Or, if you've cloned the Git mirror, run this instead:

git clean -xf python/psutil

64-bit Linux ADB for Simulator

2013-06-07T16:04:00.000-07:00

Thanks to the efforts of new Mozilla intern Brandon Kase, the latest preview build of Firefox OS Simulator for Linux includes a 64-bit version of ADB, so you can push an app to an FxOS device from 64-bit Linux installations without any extra packages!

Building it was tricky, because the Android SDK build scripts don't support that target. Brandon first tried simply specifying the target, which worked on an older version of ADB (1.0.24). But it failed on the latest version (1.0.31), which links with a bundled copy of libcrypto that includes 32-bit assembly.

Ubuntu 13.04 (Raring) ships a 64-bit android-tools-adb package, though, so we knew it could be done. And its source package's build system is much simpler than the SDK's. We just needed a binary that works on distributions with older versions of glibc than Raring's 2.17. And one that doesn't depend on a specific version of libcrypto, which varies around the Linux world; whereas Raring's ADB executable appears to need the specific version that comes with that distribution.

So Brandon modified the source package's Makefile to link libcrypto statically (note the absolute path to libcrypto.a, which may vary):

--- debian/makefiles/adb.mk    2013-03-26 14:15:41.000000000 -0700
+++ adb-static-crypto.mk    2013-06-06 16:51:52.794521267 -0700
@@ -40,15 +40,16 @@
CPPFLAGS+= -I.
CPPFLAGS+= -I../include
CPPFLAGS+= -I../../../external/zlib
+CPPFLAGS+= -I/usr/include/openssl

-LIBS+= -lc -lpthread -lz -lcrypto
+LIBS+= -lc -lpthread -lz -ldl

OBJS= $(SRCS:.c=.o)

all: adb

adb: $(OBJS)
-    $(CC) -o $@ $(LDFLAGS) $(OBJS) $(LIBS)
+    $(CC) -o $@ $(LDFLAGS) $(OBJS) /usr/lib/x86_64-linux-gnu/libcrypto.a $(LIBS)

clean:
     rm -rf $(OBJS) adb

Then I copied the source package to my CentOS 16 build machine (which has glibc 2.12) and built it there. After which the resultant executable worked on all the distributions we tested: Ubuntu 13.04, Ubuntu 10.04, CentOS 16, and Arch Linux (kernel 3.9.3-1-ARCH).

Presumably it will work on others too. But if it still doesn't work for you, let us know!

And if you just want the ADB executable, sans Simulator, here it is.

r2d2b2g implementation details

2012-10-02T12:08:00.001-07:00

Over at Mozilla Hacks, I just blogged about r2d2b2g (ratta-datta-batta-ga), an experimental prototype test environment for Firefox OS that makes it drop-dead simple to test your app in B2G Desktop.

r2d2b2g is an addon, but it bundles B2G Desktop nightly builds, which are native executables, and thus the addon is platform-specific, with packages available for Mac, Linux 32-bit, and Windows (caveat: B2G Desktop for Windows currently crashes on startup due to bug ~~794662~~ 795484).

The packages are large, 50-60MB each, partly because of the executables, but mostly because they also bundle Gaia profiles, including all default apps. (It's probably worth bundling a few of these, for demonstration purposes, but we could make the packages much smaller by removing the rest.)

r2d2b2g uses the Add-on SDK as its addon framework and relies on several third-party addon modules (subprocess, menuitems) along with some Python utilities (mozdownload, mozbase) to download and unpack B2G Desktop builds. Plus Gaia, although recent work to bundle Gaia profiles with B2G Desktop builds may break that dependency.

I've demoed the project to a variety of folks over the last couple weeks, and I've received a bunch of positive feedback about it. B2G Desktop combines approachability with phoneliness and is the best existing test environment for Firefox OS. But its configuration is a challenge, and it provides no obvious affordances for installing and testing your own app. r2d2b2g shows that these problems are tractable (even if it doesn't yet solve them all) and demonstrates a promising product path.

After seeing r2d2b2g, Kevin Dangoor drafted a PRD for a Firefox OS Simulator that I'll use to guide further development. Interested in participating? Clone the code from its GitHub repository and contribute your improvements!

Next/Previous Tab on Mac Consistent At Last

2012-03-07T18:04:00.001-08:00

After blogging about the inconsistency of keyboard shortcuts for Next/Previous Tab on Mac last year, I found out that Firefox, Thunderbird, and Komodo also support Command + Option + LeftArrow|RightArrow, and Adium has a General > "Switch tabs with" pref that I can set to the same chord.

(Later, I switched IM clients from Adium to InstantBird, which also supports that combination.)

That left Terminal, which I couldn't figure out how to configure to support the same shortcut. Until now.

I'm not sure if it's because I have since upgraded to Mac OS X 10.7 (Lion). I could've sworn I tried something like this back when I wrote that previous blog post, and it didn't work.

Go to System Preferences > Keyboard > Keyboard Shortcuts > Application Shortcuts.
Press the + (plus) button.
Select "Other..." from the Application menu and select Utilities > Terminal from the file picker dialog.
Enter "Select Next Tab" (without the quotes) into the Menu Title field.
Focus the Keyboard Shortcut field and press Command + Option + RightArrow to set the keyboard shortcut, which will appear as ⌥⌘→.
Press the Add button.

Repeat steps 4-6 with "Select Previous Tab" and Command + Option + LeftArrow, which will appear as ⌥⌘←.

Those shortcuts should now work in Terminal.

With this change, all five of my current primary productivity applications on Mac (Firefox, Thunderbird, Instantbird, Komodo, and Terminal) support a consistent pair of keyboard shortcuts for Next/Previous Tab, which are two of the most common commands I issue in all of those apps.

Woot!

generating a fingerprint for an SSH key

2012-03-07T13:46:00.001-08:00

After recently discovering a security vulnerability that allows an attacker to add an SSH key to a GitHub user account, GitHub is requiring all users to audit their SSH keys. Its audit page lists one's keys by type and fingerprint, but it doesn't say how it generated the fingerprint or how to generate one for your local copy of a key to compare it with. Nor does it let you see the whole key.

And since I don't generate such fingerprints very often, I didn't know how to do it. So I tried cksum, md5, and shasum on my Mac, but none of their checksums matched. Turns out the tool to use is ssh-keygen:

ssh-keygen -l -f path/to/keyfile

Mozilla Status Board Text is Markdown

2011-10-17T10:38:00.001-07:00

It isn't documented anywhere that I can find, but Benjamin Smedberg's handy Mozilla Status Board tool parses status text as Markdown, which is how I added a Didn't header to the Done section of my status update with all the things I planned to do last week but didn't make happen. (The Done, Next, and Coordination headers are all H4s, so I prepended four hash marks to #### Didn't to make it the same size).

(Note that [Markdown-style links](http://daringfireball.net/projects/markdown/basics) don't work and cause the entire section in which they appear to remain unparsed. However angle-bracketed URLs, as recommended by RFC 3986 <http://labs.apache.org/webarch/uri/rfc/rfc3986.html#delimiting>, work when added to the ends of lines. And "bug ###" references are auto-linkified.)

to all the bugs I've filed before

2011-09-16T08:32:00.001-07:00

The first bug I filed was marked as duplicate; the second was worksforme (although Chris Petersen could reproduce it before he couldn't anymore); and the third was invalid (it was the spec, not the code, that was errant). The fourth is the first that was fixed.

"Next/Previous Tab" Keyboard Shortcuts on Windows

2011-09-09T17:01:00.000-07:00

On my Windows laptop, I use the following four programs with tabbed interfaces on a regular basis:

Firefox
Thunderbird
Instantbird
Komodo IDE

(I'd love to have tabs in my Windows terminal app of choice, Mintty, but its developer thinks tabs should be implemented at the window manager level.)

Unlike on my Mac, all those programs implement the same keyboard shortcut for switching to the previous/next tab, and it's a simple one with just a two-key chord: Control + PageUp / PageDown.

Ha!

"Next/Previous Tab" Keyboard Shortcuts on Mac

2011-09-08T17:03:00.001-07:00

On my Mac, I use the following five programs with tabbed interfaces on a regular basis:

Firefox
Thunderbird
Adium
Terminal
Komodo IDE

And those programs implement the following five different keyboard shortcuts for switching to the previous/next tab:

Control + PageUp / PageDown (Firefox, Thunderbird)
Command + LeftArrow / RightArrow (Adium)
Command + PageUp / PageDown (Komodo IDE)
Command + Shift + [ / ] (Terminal)
Command + Shift + LeftArrow / RightArrow (Terminal)

Hrm.

gitflow vs. the SDK

2011-09-07T14:25:00.000-07:00

gitflow is a model for developing and shipping software using Git. Add-on SDK uses Git, and it too has a model, which is similar to gitflow in some ways and different in others. Here's a comparison of the two and some thoughts on why they vary.

First, some similarities: both models use multiple branches, including an ongoing branch for general development and another ongoing branch that is always ready for release (their names vary, but that's a trivial difference). Both also permit development on temporary feature (topic) branches and utilize a branch for stabilization of the codebase leading up to a release. And both accommodate the occasional hotfix release in similar ways.

(Aside: gitflow appears to encourage feature branches, but I tend to agree with Martin Fowler through Paul Julius that continuously integrating with a central development branch is preferable.)

Second, some differences: the SDK uses a single ongoing stabilization branch, while gitflow uses multiple short-lived stabilization branches, one per release. And in the SDK, stabilization fixes land on the development branch and then get cherry-picked to the stabilization branch; whereas in gitflow, stabilization fixes land on the stabilization branch and then get merged to the development branch.

(Also, the SDK releases on a regular time/quality-driven "train" schedule similar to Firefox's, while gitflow may anticipate an irregular feature/quality-driven release schedule, although it can be applied to projects with train schedules, like BrowserID.)

A benefit of gitflow's approach to stabilization is that its change graph includes only distinct changes, whereas cherry-picking adds duplicate, semi-associated changes to the SDK's graph. However, a downside of gitflow's approach is that developers must attend to where they land changes, whereas SDK developers always land changes on its development branch, and its release manager takes on the chore of getting those changes onto the stabilization branch.

(It isn't clear what happens in gitflow if a change lands on the development branch while a release is being stabilized and afterward is identified as being wanted for the release. Perhaps it gets cherry-picked?)

Overall, these models seem fairly similar, and it wouldn't be too hard to make the SDK's be essentially gitflow. We would just need to stipulate that developers land stabilization fixes on the stabilization branch, and the release manager's job would then be to merge that branch back to the development branch periodically instead of cherry-picking in the other direction.

However, it isn't clear to me that such a change would be preferable. What do you think?

Administer Git? Get a job!

2011-08-21T22:51:00.000-07:00

As I mentioned recently, Git (on GitHub) has become a popular VCS for Mozilla-related projects.

GitHub is a fantastic tool for collaboration, and the site does a great job running a Git server, but given the importance of the VCS, and because Mozilla's automated test machines don't have access to servers outside the Mozilla firewall, Mozilla should run its own Git server (that syncs with GitHub, so developers can continue to use that site for collaboration).

Unfortunately, the organization doesn't have a great deal of in-house Git server administration experience, but we're hiring systems administrators, so if you grok Git hosting and meet the other requirements, send in your resume!

Why the Add-on SDK Doesn't "Land in mozilla-central"

2011-08-11T13:33:00.000-07:00

Various Mozillians sometimes suggest that the Add-on SDK should "land in mozilla-central" and wonder why it doesn't. Here's why.

The Add-on SDK depends on features of Firefox (and Gecko), and the SDK's development process synchronizes its release schedule with Firefox's. Nevertheless, the SDK isn't a component of Firefox, it's a distinct product with its own codebase, development process, and release schedule.

Mozilla makes multiple products that interact with Firefox (addons.mozilla.org, a.k.a. AMO, is another), and distinct product development efforts should generally utilize separate code repositories, to avoid contention between the projects regarding tree management, the stages of the software development lifecycle (i.e. when which branch is in alpha, beta, etc.), and the schedules for merging between branches.

There can be exceptions to that principle, for products that share a bunch of code, use the same development process, and have the same release schedule (cf. the Firefoxes for desktop and mobile). But the SDK is not one of those exceptions.

It shares no code with Firefox. Its process utilizes one fewer branch and six fewer weeks of development than the Firefox development process, to minimize the burden of branch management and stabilization build testing on its much smaller development team and testing community. And it merges its branches and ships its releases two weeks before Firefox, to give AMO and addon developers time to update addons for each new version of the browser.

Living in its own repository makes it possible for the SDK to have these differences in its process, and it also makes it possible for us to change the process in the future, for example to move up the branch/release dates one week, if we discover that AMO and addon developers would benefit from three weeks of lead time; or to ship twice as frequently, if we determine that doing so would get APIs for new Firefox features into developers' hands faster.

Finally, the Jetpack project has a vibrant community of contributors (including both organization staff and volunteers) who strongly prefer contributing via Git and GitHub, because they find it easier, more efficient, and more enjoyable, and for whom working in mozilla-central would mean taking too great a hit on their productivity, passion, and participation.

Mozilla Labs innovates not only on features and user experience but also on development process and tools, and while Jetpack didn't lead the way to GitHub, we were a fast follower once early experiments validated its benefits. And our experience since then has only confirmed our decision, as GitHub has proven to be a fantastic tool for branch management, code review/integration, and other software development tasks.

Other Mozillians agree: there are now almost two hundred members and over one hundred repositories (not counting forks) in the Mozilla organization on GitHub, with major initiatives like Open Web Apps and BrowserID being hosted there, not to mention all the Mozilla projects in user repositories, including Rust and Zamboni.

Even if we don't make mozilla-central the canonical repository for SDK development, however, we could still periodically drop a copy of the SDK source against which Firefox changes should be tested into mozilla-central. And doing so would theoretically make it easier for Firefox developers to run SDK tests when they discover that a Firefox change breaks the SDK, because they wouldn't have to get the SDK first.

But the benefit to Firefox developers is minimal. Currently, we periodically drop a reference to the SDK revision against which Firefox changes should be tested, and developers have to do the following to initiate testing:

  wget -i testing/jetpack/jetpack-location.txt -O addon-sdk.tar.bz2
    
  tar xjf addon-sdk.tar.bz2
    
  cd addon-sdk-[revision]
    
  source bin/activate
    
  cfx testall --binary path/to/Firefox/build

We can simplify this to:

  testing/jetpack/clone
    
  cd addon-sdk
    
  source bin/activate
    
  cfx testall --binary path/to/Firefox/build

Whereas if we dropped the source instead of just a reference to it, it would instead be the only slightly simpler:

  cd testing/jetpack/addon-sdk
    
  source bin/activate
    
  cfx testall --binary path/to/Firefox/build

Either of which can be abstracted to a single make target.

But if we were to drop source instead of a reference thereto, the drops would be larger and riskier changes. And test automation would still need to be updated to support Git (or at least continue to use brittle Git -> Mercurial mirroring), in order to run tests on SDK changes, which periodic source drops do not address.

Now, this doesn't mean that no SDK code will ever land in mozilla-central.

Various folks have discussed integrating parts of the SDK into core Firefox—including stable API implementations, the module loader, and possibly the bootstrapper—to reduce the size of addon packages, improve addon startup times, and decrease addon memory consumption. I have written a very preliminary draft of a feature page describing this work, although I do not think it is a high priority at the moment, relative to the other priorities identified in the Jetpack roadmap.

And Dietrich Ayala recently suggested integrating the SDK into core Firefox for use by core features, by which he presumably also means the API implementations/module loader/bootstrapper rather than the command-line tool for testing and packaging addons.

Nevertheless, I am (and, I suspect, the whole Jetpack team is) even open to discussing integration of the command-line tool (or its replacement by a graphical equivalent), merging together the two products, and erasing the distinction between them, just as Firefox ships with core features for web development. We've even drafted a feature page for converting the SDK into an addon, which is a big step in that direction.

But until that happens, farther on up the road, the SDK is its own product that we develop with its own process and ship on its own schedule. And it has good reason to live in its own repository, and a Git one at that, as do the many (and growing number of) other Mozilla projects using similar processes and tools, which our community-wide development, collaboration, and testing infrastructure must evolve to accommodate.

SDK Training and More at Add-on-Con

2010-12-02T11:07:00.001-08:00

Next Wednesday, December 8, I'll be at Add-on-Con.

In the morning, I'll conduct a training session introducing Mozilla's new Add-on SDK, which makes it faster and easier to build Firefox add-ons. Afterwards, I'll be around and about to discuss add-ons and answer questions about the SDK and add-on development generally.

Lots of other Mozilla folks will also be on hand over the course of the two-day conference, including Dave Townsend, Jorge Villalobos, Jeniffer Boriss, Mark Finkle, and Justin Scott. A rockin' time should be had by all. Join us!

Further Adventures In Git(/Hub)ery

2010-11-27T20:47:00.001-08:00

This evening I decided to check if there were any outstanding pull requests for the SDK repository (to which I haven't been paying attention).

There were! The oldest was pull request 29 from Thomas Bassetto, which contains two small fixes (first, second) to the docs.

So I fetched the branch of his fork in which the changes reside:

$ git fetch https://github.com/tbassetto/addon-sdk.git master

But that branch (and the fork in general) is a few weeks out-of-date, so "git diff HEAD FETCH_HEAD" showed a bunch of changes, and it was unclear how painful the merge would be.

Thus I decided to try cherry-picking the changes, my first time using "git cherry-pick".

The first one went great:

$ git cherry-pick 8268334070d03a896d5c006d1b4db94d4cb44b17
Finished one cherry-pick.
[master ceadb1f] Fixed an internal link in the widget doc
1 files changed, 1 insertions(+), 1 deletions(-)

Except that I realized afterward I hadn't added "r,a=myk" to the commit message. So I tried "git commit --amend" for the first time, which worked just fine:

$ git commit --amend
[master 2d674a6] Fixed an internal link in the widget doc; r,a=myk
1 files changed, 1 insertions(+), 1 deletions(-)

Next time I'll remember to use the "--edit" flag to "git cherry-pick", which lets one "edit the commit message prior to committing."

The second cherry-pick was more complicated, because I only wanted one of the two changes in the commit (in my review, I had identified the second change as unnecessary); and, as it turned out, also because there was a merge conflict with other commits.

I started by cherry-picking the commit with the "--no-commit" option (so I could remove the second change):

$ git cherry-pick --no-commit 666ad7a99e05e338348dfc579d5b1f75e8d3bb1b
Automatic cherry-pick failed. After resolving the conflicts,
mark the corrected paths with 'git add <paths>' or 'git rm <paths>' and commit the result.
When commiting, use the option '-c 666ad7a' to retain authorship and message.

The conflict was trivial, and I knew where it was, so I resolved it manually (instead of trying "git mergetool" for the first time), removed the second change, added the merged file, and committed the result, using the "-c" option to preserve the original author and commit message while allowing me to edit the message to add "r,a=myk":

$ git add packages/addon-kit/docs/request.md
$ git commit -c 666ad7a
[master 774d1cb] Completed the example in the Request module documentation; r,a=myk
1 files changed, 1 insertions(+), 0 deletions(-)

Then I used "gitg" and "git log master ^upstream/master" to verify that the commits looked good to go, after which I pushed them:

$ git push upstream master
[git's standard obscure and disconcerting gobbledygook]

Finally, I closed the pull request with this comment that summarized what I did and provided links to the cherry-picked commits.

It would have been nice if the cherry-picked commit that didn't have merge conflicts (and which I didn't change in the process of merging) had kept its original commit ID, but I sense that that is somehow a fundamental violation of the model.

It would also have been nice if the cherry-picked commit messages had been automatically annotated with references to the original commits.

But overall the process seemed pretty reasonable, it was fairly easy to do what I wanted and recover from mistakes, and the author, committer, reviewer, and approver are clearly indicated in the cherry-picked commits (first, second).

[Also posted to the discussion group.]

More Git/Hub Workflow Experiences

2010-11-27T20:39:00.001-08:00

After posting about my first Git/Hub workflow experiences, I got lots of helpful input from various folks, particularly Erik Vold, Irakli Gozalishvili, and Brian Warner, which led me to refine my process for handling pull requests:

From the "how to merge this pull request" section of the pull request page (f.e. pull request 34), copy the command from step two, but change the word "pull" to "fetch" to fetch the remote branch containing the changes without also merging it:

git fetch https://github.com/toolness/jetpack-sdk.git bug-610507
Use the magic FETCH_HEAD reference to the last fetched branch to verify that the set of changes is what you expect:

git diff HEAD FETCH_HEAD

(The exact syntax here may need some work; HEAD..FETCH_HEAD? three dots?)
Merge the remote branch into your local branch with a custom commit message:

git merge FETCH_HEAD --no-ff -m"bug 610507: get rid of the nsjetpack package; r=myk"
Push the changes upstream:

git push upstream master

I like this set of commands because it doesn't require me to add a remote, I can copy/paste the fetch command from GitHub (being careful not to issue the pull before I change it to a fetch), and I always type the same FETCH_HEAD reference to the remote branch in step three.

However, I wish the merge commit page explicitly referenced the specific commits that were merged. It does mention that it's a branch merge, it isn't obvious how to get from that page to the pages for the commits I merged from the branch.

"git log --oneline --graph", gitg, and gitk do give me that information, though, so I'm ok on the command line, anyway.

[More discussion can be found in the discussion group thread.]

Git/Hub Workflow Experiences

2010-11-12T18:55:00.000-08:00

The Jetpack project recently migrated its SDK repository to Git (hosted on GitHub), and we've been working out changes to the bug/review/commit workflow that GitHub's tools enable (specifically, pull requests).

Here are some of my initial experiences and my thoughts on them (which I've also posted to the Jetpack discussion group).

Warning: Git wonkery ahead, with excruciating details. I would not want to read this post. I recommend you skip it. ;-)

Part 1: Wherein I Handle My First Pull Request

To fix some test failures, Atul submitted GitHub pull request 33, I reviewed the changes (comprising two commits) on GitHub, and then I pushed them to the canonical repository via the following set of commands:

git checkout -b toolness-4.0b7-bustage-fixes master
git pull https://github.com/toolness/jetpack-sdk.git 4.0b7-bustage-fixes
git checkout master
git merge toolness-4.0b7-bustage-fixes
git push upstream master

That landed the two commits in the canonical repository, but it isn't obvious that they were related (i.e. part of the same pull request), that I was the one who reviewed them, or that I was the one who pushed them.

Part 2: Wherein I Handle My Second Pull Request

Thus, for the fix for bug 611042, for which Atul submitted GitHub pull request 34, I again reviewed the changes (also comprising two commits) on GitHub, but then I pushed them to the canonical repository via this different set of commands (after discussion with Atul and Patrick Walton of the Rust team):

git checkout -b toolness-bug-611042 master
git pull https://github.com/toolness/jetpack-sdk.git bug-611042
(There might have been something else here, since the pull request resulted in a merge; I don't quite remember.)
git checkout master
git merge --no-ff --no-commit toolness-bug-611042
git commit --signoff -m "bug 611042: remove request.response.xml for e10s compatibility; r=myk" --author "atul"
git push upstream master

Because Atul's pull request was no longer against the tip (since I had just merged those previous changes), when I pulled the remote bug-611042 branch into my local toolness-bug-611042 branch (step 2), I had to merge his changes, which resulted in a merge commit.

Merging the changes to my local master with "--no-ff" and "--no-commit" (step 5) then allowed me to commit the merge to my master branch manually (step 6), resulting in another merge commit.

For the second merge commit, I specified "--signoff", which added "Signed-off-by: Myk Melez " to the commit message; crafted a custom commit message that included "r=myk"; and specified '--author "atul"', which made Atul the author of the merge.

I dislike having the former merge commit in history, since it's extraneous, unuseful details about how I did the merging locally before I pushed to the canonical repository. I'm not sure how to avoid it, though.

On the other hand, I like having the latter merge commit in history, since it provides context for Atul's two commits: the bug number, the fact that the changes were reviewed, and a commit message that describes the changes as a whole.

I'm ambivalent about --signoff vs. adding "r=myk" to the commit message, as they seem equivalentish, with --signoff being more explicit (so in theory it might form part of an enlightened workflow in the future), while "r=myk" is simpler.

And I dislike having made Atul the author of the merge, since it's incorrect: he wasn't the author of the merge, he was only the author of the changes (for which he is correctly credited). And if the merge itself caused problems (f.e. I accidentally backed out other recent changes in the process), I would be the one responsible for fixing those problems, not Atul.

Part 3: Pushing Patches

In addition to pull requests, one can also contribute via patches. I've pushed a few of these via something like the following set of commands:

git apply patch.diff
git commit -a -m "bug : ; r=myk" --author ""
git push upstream master

That results in a commit like this one, which shows me as the committer and the patch author as the author. And that seems like a fine record of what happened.

Part 4: To Bug or Not To Bug?

One of the questions GitHub raises is whether or not every change deserves a bug report. And if not, how do we differentiate those that do from the rest?

I don't have the definitive answers to these questions, but my sense, from my experience so far, is that we shouldn't require all changes to be accompanied by bug reports, but larger, riskier, time-consuming, and/or controversial changes should have reports to capture history, provide a forum for discussion, and permit project planning; while bug reports should be optional for smaller, safer, quickly-resolved, and/or non-controversial changes.

My Recent Jetpack Presentations

2010-07-15T14:22:00.000-07:00

The last few weeks have been presentation-heavy.

First, I gave a presentation about the Jetpack project (past accomplishments, present status, future plans) at the 2010 London Mozilla Add-ons Workshop (MAOW), including a demo of using Add-on Builder to build an add-on in five minutes.

Then I reprised the Add-on Builder demo as part of the opening day keynote at the Mozilla Summit, where it got a great reception. You can watch it in this Youtube video.

Finally, I gave an updated version of the MAOW presentation on the third day of the summit. The slides are available in OpenDocument and PDF formats, and Jetpack presentation materials generally are all available from the Jetpack Presentations wiki page.

This blog has moved

2010-03-05T23:03:00.001-08:00

This blog is now located at http://mykzilla.blogspot.com/.
You will be automatically redirected in 30 seconds, or you may click here.

For feed subscribers, please update your feed subscriptions to
http://mykzilla.blogspot.com/feeds/posts/default.

The Skinny on Raindrop's Mailing List Extensions

2009-11-17T17:26:00.000-08:00

Raindrop is an exploration of messaging innovation that strives to intelligently assist people in managing their flood of incoming messages. And mailing lists are a common source of messages you need to manage. So, with assistance from the Raindrop hackers, I wrote extensions that make it easier to deal with messages from mailing lists.

Their goal is to soothe two particular pain points when dealing with mailing lists: grouping their messages together by list and unsubscribing from them once you're no longer interested in their subject matter.

This post explains how the extensions do this; touches on some aspects of Raindrop's message processing and data storage models; and speculates about possible future directions for the extensions.

Raindrop Extensibility

Raindrop is being built with the explicit goal of being broadly and deeply extensible, and it includes a number of APIs for adding and modifying functionality. The mailing list enhancements comprise two related extensions, one in the backend and one in the user interface.

The backend extension plugs into Raindrop's incoming message processor, intercepting incoming email messages and extracting info about the mailing lists to which they belong. It also handles much of the work of unsubscribing from a list.

The frontend extension plugs into Raindrop's Inflow application, modifying its interface to show you the most recent mailing list messages at a glance, group mailing list conversations together by list, and provide a button you can press to easily unsubscribe from a mailing list.

Message Processing and Data Storage

Before getting into how the extensions work, it's useful to know a bit about how Raindrop processes and stores messages.

Raindrop stores information using CouchDB, a document-centric database whose principal unit of information storage and retrieval is the document (the equivalent of a record in SQL databases). Documents are just JSON blobs that can contain arbitrary name -> value pairs (unlike SQL records, which can only contain values for predeclared columns).

To distinguish between different kinds of documents, Raindrop assigns each a schema (similar to a table in SQL parlance) that describes (and may one day constrain) its properties. The rd.msg.email schema is the primary schema representing an email message, while the rd.mailing-list is the schema representing a mailing list, and the rd.msg.email.mailing-list is a simple schema that associates messages with their lists.

(In an SQL database, rd.msg.email and rd.mailing-list would be tables whose rows represent email messages and mailing lists, while rd.msg.email.mailing-list would be a table whose rows map one to the other.)

Note that there's a many-to-one relationship between messages and lists, since messages belong to a single list, although lists contain many messages, so rd.msg.email.mailing-list isn't strictly necessary. Its list-id property (which identifies the list to which the message belongs) could simply be a property of rd.msg.email docs (or, in SQL terms, a foreign key in the rd.msg.email table).

But putting it into its own document has several advantages. First, it improves robustness, as it reduces the possibility of conflicts between extensions and core code writing to the same documents.

It also improves write performance, as it's faster to add a document than to modify an existing one (although index generation and read performance can be an issue).

Finally, it improves extensibility, because it makes it possible to write an extension that extends the backend mailing list extension.

That's because Raindrop's incoming message processing model allows extensions to observe the creation of any kind of document, including those created by other extensions.

So just as the mailing list extension observes the creation of rd.msg.email documents, another extension can observe the creation of rd.msg.email.mailing-list documents and process them further in some useful way. If the mailing list extension simply modified the original document instead of creating its own, that would require some additional and more complicated API.

The Backend Extension

The primary function of the backend extension is to examine every incoming message and dress the ones from mailing lists with some additional structured information that the frontend can use to organize them.

Backend extensions are accompanied by a JSON manifest that tells Raindrop what kinds of incoming documents it wants to intercept. The mailing list extension's manifest registers it as an observer of incoming rd.msg.email documents, which get created when Raindrop retrieves an email message:

"schemas" : {
  "rd.ext.workqueue" : {
      "source_schemas" : ["rd.msg.email"],
...

The extension itself is a Python script with a handler function that gets passed the rd.msg.email document and looks to see if it contains a List-ID header (or, in certain cases, another identifier) identifying the mailing list from which the message comes:

def handler(message):
  ...
  if 'list-id' in message['headers']:
      # Extract the ID and name of the mailing list from the list-id header.
      # Some mailing lists give only the ID, but others (Google Groups,
      # Mailman) provide both using the format 'NAME <id>', so we extract them
      # separately if we detect that format.
      list_id = message['headers']['list-id'][0]
  ...

If it doesn't find a list identifier, it simply returns, and Raindrop continues processing the message:

if not list_id:
    logger.debug("NO LIST ID; ignoring message %s", message_id)
    return

Otherwise, it calls Raindrop's emit_schema function to create an rd.msg.email.mailing-list document linking the message document to an rd.mailing-list document representing the mailing list:

emit_schema('rd.msg.email.mailing-list', { 'list_id': list_id })

In this function call, rd.msg.email.mailing-list is the type of document to create, while { 'list_id': list_id } is the document itself, written as Python that will get serialized to JSON.

A document created inside a backend extension like this automatically gets a reference to the document the extension is processing (i.e. the rd.msg.email document), so the only thing it has to explicitly include is a reference to the list document, in the form of a list_id property whose value is the list identifier.

The extension also checks if there's an rd.mailing-list document in the database for the mailing list itself, and if not, it creates one, populating it with information from the message's List-* headers, like how to unsubscribe from the list. Otherwise, it updates the existing mailing list document if the message's List-* headers contain updates.

The Frontend Extension

The frontend extension uses the information extracted by the backend to help users manage mailing lists in the Inflow application.

It adds a widget to the Home view that shows you the last few messages from your lists at the bottom of the page, so you can keep an eye on those messages without having to give them your full attention:

It adds a list of your mailing lists to the Organizer widget:

And when you click on the name of a list, it shows you its conversations in the conversation pane:

In traditional mail clients, users who want to break out their list messages into separate buckets like this typically have to create a folder for each list to contain its messages and then a filter for each list to move incoming list messages into the appropriate folders. The extension does this for you automatically!

Finally, while viewing list conversations, if the extension knows how to unsubscribe you from the list, it displays an Unsubscribe button:

Pressing the button (and then confirming your decision) unsubscribes you from the list. You don't have to do anything else, like remembering your username/password for some web page, sending an email, or confirming your request with the list admin. The extensions handle all those details for you so you don't have to know about them!

List Unsubscription

In case you do want to know the details, however, it goes like this...

First, the frontend extension sends a message to the list's admin address requesting unsubscription, with a certain command (like "unsubscribe") in the subject or body of the message (lists often specify exactly what command to send in the mailto: link they include in the List-Unsubscribe header):

From: Jan Reilly 
To: wasbigtalk-admin@example.com
Subject: unsubscribe

Then the server responds with a message requesting confirmation of the request, often putting a unique token into the Subject or Reply-To header to track the request:

From: wasbigtalk-admin@example.com
To: jan@example.com
Subject: please confirm unsubscribe from wasbigtalk (4bc3b7e439fd)

Hello jan@example.com,

We have received a request to unsubscribe you from wasbigtalk.
Please confirm this request to unsubscribe by replying to this email.
...

Then the backend extension responds with a message confirming the request that includes the unique token:

From: jan@example.com
To: wasbigtalk-admin@example.com
Subject: Re: please confirm unsubscribe from wasbigtalk (4bc3b7e439fd)

Finally, the server responds with a message confirming that the subscriber has, indeed, been unsubscribed:

From: wasbigtalk-admin@example.com
To: jan@example.com
Subject: you have been unsubscribed from wasbigtalk

Hello jan@example.com,

Your unsubscription from wasbigtalk was successful.
...

At this point, the backend extension marks the list unsubscribed in the database, and the frontend extension marks it unsubscribed in the user interface.

This process matches the way much mailing list server software works, although there are daemons in the details, so the extensions have to be programmed to support each server individually.

Currently, they know how to handle Google Groups and Mailman lists. Majordomo2 (used by the Bugzilla and OpenBSD projects, among others) is not supported, because it doesn't send List-* headers (alhough supposedly it can be configured to do so). The W3C's list server is not yet supported, although it does send List-* headers, and support should be fairly easy to add.

Note that some of the processing the extension does is (locale-dependent) "screen"-scraping, as Google Groups and Mailman don't consistently identify the list ID and message type in some of their correspondence. In the long run, hopefully server software will improve in that regard. Perhaps someone can spearhead an effort to make it so?

The Future

The extensions' current features fit in well with Raindrop's goal of helping people better handle their flood of incoming messages. But there is surely much more they could do to help in this regard.

Besides general improvements to reliability and robustness--like support for additional list servers and handling of localized admin messages--they could let you resubscribe to a mailing list from which you've unsubscribed. And perhaps they could automatically fetch the messages you missed while you were away. Or even retrieve the entire archive of a list to which you're subscribed, so you can browse the archive in Raindrop!

What bugs you about mailing lists? And how might Raindrop's mailing list extensions make them easier (and even funner) to use?

Building/Releasing Personas

2009-11-04T15:58:00.001-08:00

Want to know how a popular extension like Personas gets built and released? Neither do I! Yet I know anyway. And I've written it down for your edification! So check it out.