A few months ago I joined up with the other OSD700 students on the WebVTT feature for Firefox. Its purpose is display captions and subtitles for videos using the Track element. The main problem was that I wasn’t on the initial team that started the parser in OSD600, so I was a bit behind on what needed to be done. So I ended up choosing a portion of this feature that wasn’t directly related to the WebVTT parser.
I started out with a simple layout-related task. Add an anonymous div to the video element for the caption overlay. A bit of unfinished code was already done for this as a prototype by rillian (Ralph Giles). That prototype was unfortunately a year old, and a lot had changed in the codebase since that time. So, once things were up to date, I opened an issue on Bugzilla for it. Thankfully due to the bug’s specific nature, it only took some style changes and updating to actually get approval by roc (Robert O’ Callahan). Mozilla seems to have very strict rules on whitespace, and keeping style consistent, and I can admire that. Consistency definitely helps for readability.
And so I waited until it was time to get things together. A lot of parts started depending on each other to get a working prototype. So to rush for a prototype to show off, we started integrating our pieces, using Rick Eyres integration branch. We found out that getting all these things merged together and actually building would be an annoying task. Eventually, we ended up with something we could show. I changed up the css for the caption overlay so that it would be positioned correctly, and a little easier to look at, and we were ready to go. I added a text shadow instead of a black background, changed the font family/size, and changed the text colour to white.
We went to the Mozilla offices to show it off, and several of the students presented their own work on the feature. Several people from the Mozilla Toronto office also presented some of their own projects for the future of the web that looks very exciting. Some of the work on asm.js and WebRTC looks and sounds great.
Recently, I was tasked with giving another child div to each Text Track. The reasoning behind it was that positioning in the prototype wasn’t done properly (it would show below the video in 16:9 ratio). It used padding-top: 80% . While that would work when you knew the size of the video, it wouldn’t work any other time. If there was another child div, you could use the CSS trick where the outer div is relatively positioned, while the inner div is absolute positioned. That would allow you to place the inner div at the bottom of the outer one easily with bottom: 0, or an amount of exact pixels up you wanted it to be. So I positioned it just above the controls. This inner child div also allowed the caption itself to possibly be clickable, but the outer div to not be clickable.
I put these changes where the cue text was being appended to the caption overlay in WebVTTLoadListener. I could already see that was the wrong place to do that, but I might as well get things working first. I opened up a pull request to integration to see what I could change before it got in. There was an unforeseen issue with getting the pull request itself up. The fork of mozilla-central that I had from a year ago on github wouldn’t do pull requests anymore, it would just give me a 500 error. I tried waiting, I tried asking for help, but nobody seemed to have an answer to this issue. Eventually, I just renamed the repo, and forked another repository of mozilla-central on github instead. Then I pushed up my work again. This fixed it. For this patch, I put a member in TextTrack for the child div, and appended it as the cue text was appended. As I thought, it was the wrong place to be doing this. Rick Eyre was already preparing a patch that moved a lot of this logic to TextTrackCue. Once that was in, I put my patch for the child divs back in, this time as a member of TextTrackCue. With that working, it gets merged in, and we’re one step closer to bringing everything together.
So I needed to be able to accurately pick at things in a 3D scene in CubicVR, but a bounding box ray test (scene.bbRayTest()) wasn’t providing that. If a mesh had an odd shape, or holes, it wouldn’t be very accurate, since everything was picked based on a bounding box. So I looked up online how other people were doing it. The general answer was called colour picking. The general process for this would be:
1. Render each pickable object to a backbuffer with a slightly different colour.
2. Get the colour at the pixel your mouse is at.
3. See if it matches up with anything in the scene that is pickable. If so, you have a match.
It seems simple enough, but there’s a lot of little things that needed to be taken care of before it would actually work.
– It needed to be flat shaded, no lights at all.
– Nothing should affect the color (such as texture, wireframe mode, ambient, specular, etc..). So the material has to be completely different, and wireframe mode on the scene object needs to be set off.
– CubicVR stores colors as floats between 0 and 1, gl.readPixels() gives you bytes. You need to take that into account.
– The mouse position y you send to gl.readPixels() has to be flipped, so you use height – mouseY instead.
To test this, I took the CubicVR picking demo that already existed, and tried to use my colour picking code instead of scene.bbRayTest(). Click here to view this test, and here to view the original test. It’s amazing how something simple can have a lot of edge cases you need to take care of before anything will work.
Edit: It seems that in the example, I actually had a couple of issues that I still didn’t catch. Mainly that I was doing the picking code in the main loop, creating the backbuffer every check, and creating a new material for each object every check. Firefox didn’t like this if left alone. So, I cached both the unique material, and the drawing buffer. Then I only called the picking code on movemove, an recreated the buffer during window resize. That should solve the earlier issues I had.
Thankfully now we’re at the point where we have most of the Mouse Lock feature set from the specifications implemented, so that’s most of the work done already. Now we have to make sure that everything stays working, and that all edge cases and bugs are properly dealt with.
Last Sunday, I converted the CubicVR FPS Demo to use Mouse Lock. All of our current demos are up over here, if you have one of our current builds with mouse lock in it. It showed it off pretty well how Mouse Lock in the browser can work for a game.
Now I also have to write a few tests, mainly to see if pointer.lock() and pointer.unlock() are working as they should under different conditions, and that the success and failure callbacks from pointer.lock() are firing at the right times. So I looked at the current mouselock mochitest example, which had some of this already done (mouselock/test_fullscreen.html). The first thing I saw was that it was trying to work off a setTimeout for mozRequestFullScreen(), instead of doing it on load, so I changed that. I ended up doing most of my logic inside the “mozfullscreenchange” event, since it’s pretty easy to check for different things relating to mouse lock in there. So I’m doing different tests, and find out randomly that pointer.lock(null) will actually break the browser in the current build. So I look at the stack trace the test gave me, and luckily it showed the exact line number/file it happened in.
nsCOMPtr<nsIDOMNode> targetNode(do_QueryInterface(aTarget));
nsCOMPtr<nsIDOMNode> parentNode;
targetNode->GetParentNode(getter_AddRefs(parentNode)); // This is going to crash if targetNode is NULL.
Therefore, before targetNode is deferenced, I had to check for targetNode being NULL, just a simple
if (!targetNode) return false;
I rebuilt and tested for the same thing, and now it doesn’t crash.
Hooray for testing!
So yesterday I had a working windows version of mouse lock. The problem was that it required you to go a long way just to get both of the things you needed for the widget to know mouse lock was on, and it was OS specific. But the new solution says that nsIWidget doesn’t need to know.
nsEventStateManager will store if mouse lock is on. And the MouseLockable on lock() and unlock() will set that boolean value. When the mouse is moved and mouse lock is on, the nsEventStateManager will call aEvent->widget->SynthesizeNativeMouseEvent() with the coordinates of center of the screen, and the appropriate native message based on the OS. nsIWidget’s SynthesizeNativeMouseEvent() calls SetCursorPos() on windows just like I was doing, and the correct version for each OS. So it looks like everything is still working, but with a lot less code, and a lot more code that works on all supported operating systems.
The only problem is that not all operating systems have nsIWidget’s SynthesizeNativeMouseEvent() function implemented, so we’re going to have to do some research for the ones that don’t (for example, the GTK2 Linux one was oddly absent), and implement it ourselves.
So, with Humph showing me nsEventStateManager, a lightbulb went off in my head. nsEventStateManager::GenerateMouseEnterExit(aEvent) takes an nsGUIEvent* as its parameter. nsGUIEvent has a widget attached to it. We can get the widget now. Then I found out about nsFocusManager::GetFocusManager()->GetFocusedWindow(ptr), which gives you the nsIDOMWindow currently in focus. Great! Now I can keep going down the tree until I find my MouseLockable instance. Then, all I need to do to tell the widget that mouse lock is now on is to set a boolean in it. Now that the nsIWidget (nsWindow is one), knows that its mouse lock is on or off, it can do OS-Specific code for it, as described a few posts back.
(In GenerateMouseEnterExit(aEvent)) http://pastebin.com/pt0u2vdr
I also ran into a little bug with the DOM’s movement deltas when centering the mouse cursor every mouse move while in mouse lock. (it kept reporting opposite values). The solution was actually simple. Just report the last mouse position value as width/2, height/2, just like what I set the cursor too. If it’s anything but that, it must have moved from the center.
No clue if anything is in the right place of course. But it feels great to have some stuff actually working.
Time to test this with some games!
What I did was added a non-scriptable method to an nsIDOMWindow that saves the X/Y so that the next mouse event could get it, as well as attributes for the previous x/y.
(in nsIDOMWindow.idl)
[noscript] void saveScreenPosition(in long x, in long y);
readonly attribute long lastScreenX;
readonly attribute long lastScreenY;
Then, in nsGlobalWindow, that function simply sets the two members. When an nsDOMMouseEvent is created, it stores the movement deltas based on the previous/current values, then tells the nsGlobalWindow to save the current one.
(In nsDOMMouseEvent’s constructor, mView is the nsIDOMWindow)
if (mView) {
PRInt32 screenPosX, screenPosY, lastScreenPosX, lastScreenPosY;
GetScreenX(&screenPosX);
GetScreenY(&screenPosY);
mView->GetLastScreenX(&lastScreenPosX);
mView->GetLastScreenY(&lastScreenPosY);
mMovement.x = screenPosX – lastScreenPosX;
mMovement.y = screenPosY – lastScreenPosY;
mView->SaveScreenPosition(screenPosX, screenPosY);
}
And that was pretty much it, success.
I shall talk about the stuff I did regarding Perlin Noise in Gladius and my adventures through that tomorrow, it was pretty fun. And I’ll be trying to do make the mouse cursor disappear tonight as we talked about in class today. I’m starting to see how more stuff is coming together, getting widgets from the DOM window is a step in the right direction in learning where things connect. I’m frustrated how many times I’m failing my builds because I get something small wrong and have to start a 6-10 minute process all over again. But what can you do, there’s a lot of stuff to compile and link at the top-level.
I’ve been looking through the Firefox source in an attempt to find out how to make some of these components talk to each other, but I keep on running into some sort of issue preventing me from doing so. A couple days back, I pulled Humph’s latest revision which had the MouseLockable interface, and had that put onto window.navigator.pointer. I’m trying to figure out a way for the OS-Specific code in nsWindow to know about the MouseLockable reference in Navigator, or for the MouseLockable to be able to tell the nsWindow to start/stop doing mouse lock. Once that is possible, my previous solution should be able to work properly, hopefully. The specification also states that the MouseLockable should only be able to have Mouse Lock turned on while FullScreen is enabled, it shouldn’t be that hard if it has a pointer to an nsIDOMWindow (like nsGeoLocation), which stores that information.
As well, I followed another post which describes how to add movement into the MouseEvent interface. That’s all well and good until you find out that no where actually stores Mouse X/Y. It simply wasn’t built for it. Calling GetScreenPoint() in the event goes to the OS native code via the widget and gets the position directly and puts it inside refPoint. That’s all well and good, but events are created and destroyed in the blink of an eye, you can’t store previous mouse coordinate values there. So, seeing as the event stores an nsIBaseWidget, and nsWindow is one, it seems like previous/current mouse coordinate values should be stored there. That’s fine until I remembered that pointer is to an nsIBaseWidget, and I don’t know how to detect the type to cast it up safely to get those values. So I’m at a bit of a dead end here, hopefully tomorrow’s class can clear some things up.
Since the code base is so large it’s very easy to get lost, and I still feel pretty lost in all of this myself. Within the next few weeks I hope a lot of things can be cleared up. I think a lot of us feel this way right now.
After a good amount of researching through the Firefox source, especially with the Cross Reference website I’ve found out how to get mouse lock sort-of working on Windows. The key to this was the ClipCursor function in windows.h. This takes a rectangle which says where the mouse cursor can move around in. By setting this to the windows width/2, height/2, for the left/top, and the right/bottom, you’ll have a rectangle with no width and height in the middle of the screen, with a cursor that cannot move.
Here’s my little snippet that does just that:
The only problem was, this doesn’t track mouse deltas, it doesn’t seem to allow for it. So I looked up the Win32 API some more on how to set the mouse position directly. I found SetMouseCursor(x,y), perfect. So my new logic was, if the mouse is not in the center, it must have moved. If it has moved, track the movement change then force it back into the center. It seems to have worked, but I still have to test it a bit more.
So I ended up with this (requires adding the mMousePos and mMouseLock member variables):
I have no clue if this is right, but I’m guessing it’s a step in the right direction in terms of what to do on Windows. Next time I’ll have to research more about getting the mouse lock/unlock functions visible on the DOM.
While in class, Humph showed us how to get some output going by changing parts of the firefox source. I simply copied where he put the test printf, tried to build it, and saw the results. Oddly enough, my top-level build ended up taking me 20 minutes to complete. But when I ran firefox, the printf showed up fine, several times. Next up, to try to fiddle around with the way the mouse works in firefox, and to keep seeing what I can change in different parts of the code.
Jesse Silver's Blog 