Final Boss release 0.3

For the final release, I wanted to work on multiple small issues on a project I had some familiarity.  I wanted to try to do more issues this time and apply the knowledge I have learned throughout this semester.  From the topics we touched on in class, such as TravisCI, the bridge-troll project, to opening PDFs in Brave browser, I wanted to use the things I learned and contribute that way.  This was why I wanted to work on projects I had some experiences with.  The first project I chose was the Brave-laptop which I had built and worked on during previous labs.

Issue 1:

The first issues I searched for were from the url bar labels, and I found the following to be of interest.  The issue was that ‘foo.b’ were causing brave to go to http://foo.b/ instead of performing a search for ‘foo.b’ like the other browsers.  I thought I would have a good handle on this issue, because of my experience in handling url query with ‘dog cat’ and ‘http://www.google.ca?q=dog cat’ from the previous lab.  I knew that if I fiddled with the urlutil.js file and its regular expression for awhile, I would be able to fix this issue.  However, as I dove deeper into the comments from the conversation between the reporter and the contributors of the community, I realized the issue was more in depth than it first appeared.  The issue lied in the autocomplete function of the url address bar rather than how the url was processed.  Specifically, they wanted ‘github.c’ to autocomplete to ‘github.com’, from the references in this conversation.  It appeared that the person who reported the issue was working directly with the owner of the project already, and so I continued my search for other issues.

This slideshow requires JavaScript.

Issue 2:

The second issue I explored involved opening pdf files in the brave browser.  The bug in this issue was opening PDFs file locally would result in a blank pdf being opened. The first step is to download a pdf from a site.  The issue recommended to grab the pdf from here, and store it onto the local computer.  Afterwards, attempt to open this pdf in Brave under ‘File > Open File > (select to pdf file)’.  The result is a blank page as per the screenshot below.

This slideshow requires JavaScript.

My first thought to this issue was to apply my professor, David Humphrey’s fix to opening the pdf files in Brave.  He posted a case study where he goes through in detail the steps and thought process to fixing the pdf issue he encountered.  Unlike the issue I found, his issue involved html files which had pdfs embed in them, with urls that ended with the ‘.pdf’ extension.  Unfortunately, after applying his fix removing the 3 lines in urlutil.js, I was still not able to open the pdf files locally.

The fix for embedded PDFs was to remove the three lines in toPDFJSLocation(). I added some extra code to decode the URI and make sure it was not the cause of my failure to open pdfs locally

The next thing I looked at was the function in the ‘tab.js’ file called fixDisplayURL().  This function populates the field in a navigationEntry object, and calls getLocationIfPDF() to check if the PDFUrl is a valid and then return the query file Url.  This means that the PDFUrl has to start with the the chrome base extension.  For example:

Input PDFUrl:

chrome-extension://jdbefljfgobbmcidnmpjamcbhnbphjnb/content/web/viewer.html?file=file%3A%2F%2F%2Fhome%2Fomak%2FDocuments%2Ftest.pdf

returned query Url:

file:///home/omak/Documents/test.pdf

Afterwards, getLocationIfPDf() searches for the string, ‘content/web/viewer.html?file=’ and if it finds this string, it will extract the query following this string, and return it back to fixDisplayUrl().   The following screenshot shows what is passed into getLocationIfPDF(), and the returned url from the function:

This is to check if fixDisplayUrl is causing errors by incorrectly passing the urls to other functions, which may result in the blank pdf

From the results from my log statements, I could not find anything wrong with fixDisplayUrl(), as it seems to be working properly.  One of my concern was the ‘origin’ member of the navigationEntry object, as it displayed the url without the ‘content/web/viewer.html?file=’  in it, but after fiddling with the member to ensure the missing part of the string was added, it had no impact on the behaviour of the issue I was facing.  The following is a gist of my work on the fixDisplayUrl:

	// hack to deal with about:* pages
	const fixDisplayURL = (navigationEntry, controller) => {
	if (navigationEntry == null) {
	return null
	}
	navigationEntry = Object.assign({}, navigationEntry)
	//console.log("fixDisplayURL–>IN–>", navigationEntry.virtualURL)
	navigationEntry.virtualURL = getLocationIfPDF(navigationEntry.virtualURL)
	//console.log("fixDisplayURL–>OUT–>", navigationEntry.virtualURL)
	if (isTargetAboutUrl(navigationEntry.virtualURL)) {
	navigationEntry.virtualURL = getSourceAboutUrl(navigationEntry.virtualURL)
	}
	if (isIntermediateAboutPage(navigationEntry.virtualURL) &&
	!navigationEntry.virtualURL.startsWith('about:safebrowsing#')) {
	const previousEntry = controller.getEntryAtOffset(-1)
	if (!controller.canGoForward() && previousEntry) {
	navigationEntry.virtualURL = previousEntry.virtualURL
	}
	}
	if (isTargetMagnetUrl(navigationEntry.virtualURL)) {
	navigationEntry.virtualURL = getSourceMagnetUrl(navigationEntry.virtualURL)
	}
	if (navigationEntry.virtualURL === 'about:newtab') {
	navigationEntry.virtualURL = ''
	}
	navigationEntry.virtualURL = muon.url.formatForDisplay(navigationEntry.virtualURL)
	//console.log("fixDisplayURL2–>", navigationEntry)
	const parsedURL = muon.url.parse(navigationEntry.virtualURL)
	//console.log("fixDisplayURL3–>", parsedURL)
	navigationEntry = Object.assign(parsedURL, navigationEntry)
	if(navigationEntry.url.startsWith('chrome-extension://jdbefljfgobbmcidnmpjamcbhnbphjnb')){
	//navigationEntry.url = 'chrome-extension://jdbefljfgobbmcidnmpjamcbhnbphjnb/content/web/viewer.html?file=' + navigationEntry.virtualURL
	//navigationEntry.path = '/test.pdf';
	//navigationEntry.pathname = '/test.pdf';
	//navigationEntry.origin = navigationEntry.virtualURL;
	}
	//console.log("fixDisplayURL4–>", navigationEntry)
	return navigationEntry
	}

view raw

tab.js

hosted with ❤ by GitHub

After an hour or two fiddling with both the getLocationIfPDF() function and the fixDisplayUrl() function, I shifted my focus towards the file pdfJS.js.  There are 3 functions of note:

• onBeforeRequest()
• onHeadersReceived()
• isPDFFile()
• isPDFDownloadable()

One of the first things I wanted to find out was the different between loading a pdf from a URL and loading a pdf from a local file path.  The first thing I found out was that with a url path such as ‘http://www.orimi.com/pdf-test.pdf’, onBeforeRequest() is called and processed, afterwards onHeadersReceived() is fired and checks if it uses isPDFFile() and isPDFDownloadable() to check and retrieve the pdf file.  It will look for things such as the ‘Content-type:’ being ‘application/pdf’ before rendering the pdf file.  However, when I pass in a local file path such as ‘file:///home/omak/Documents/test.pdf’, only the onBeforeRequest() function is triggered.

This slideshow requires JavaScript.

The onHeadersReceived() function is never called when opening a local file.  This is likely caused by the line: appActions.loadURLRequested(details.tabId, getViewerUrl(details.url))

This line opens the Url that is passed into the function, and the browser will render that url.  From my testing, it never returns from this call.  This means the subsequent lines such as ‘return result’ are never executed, and onHeadersReceived() is never called.  UPDATE:  Upon further testing, my hypothesis was not correct, and the subsequent lines were called, but onHeadersReceived() was still not triggered.

To investigate this matter further, my professor suggested that I use the ‘console.trace()’ command to see if I figure out the call stacks for onBeforeRequest().  This led me to the filtering.js, which calls onBeforeRequest(), and stores the results in an array used for registering headers later on.

Stack trace using console.trace() to find out the where functions were called from. This led me to filtering.js

This led me to explore the filtering.js file, which was filled with unfamiliar code that dealt with handling appStates, appActions, and muonCb, all of which I could not understand.  In particular, I wanted to find out what happens when ‘result.cancel’ is set to be true, which was the case for opening a local pdf file.  Within filtering.js, there is a section of code that dealt of this case, but I couldn’t figure what it was doing in its entirety.

From further testing where I commented out the ‘result.cancel = true’ line in pdfJS.js, the behaviour for opening a local file is that the browser will prompt to re-download this file.  The behaviour will be shown on the screenshot below.

However, if I only commented out the ‘appActions.loadURLRequested(details.tabId, getViewerUrl(details.url))’ line here, the resulting behaviour is that nothing happens, and I only receieve error page.

This slideshow requires JavaScript.

After all of the above attempts, I thought it would be best if I search for another bug to fix instead, since I could not go much further. Since I was not able to go further, I am posting my work here within this branch of my fork in hopes that I can maybe solve this another time.

I switched back to bug hunting in the VSCode project instead and found some issues I could handle.

Issue 3:

Since I’ve burned a lot of my allotted time on my second attempt, I decided to find something smaller to fix this time around. My classmate, Matthew Quan, suggested that a previous issue mentioned in class earlier in the semester was still not fixed.  The issue is located here: https://github.com/Microsoft/vscode/issues/42726.

The problem was in vscode and how it searches absolute paths via the quick open (ctrl+P) command.  If one of the directories names or file names included a ‘space’ in the string, quick open would not be able to open that file.  Back in February, our professor had created a solution to this problem, and posted this walkthrough.  He would check for a path separator in the file path, and then remove in the file path if it contained a path separator.  He was about to submit a PR for the issue until another classmate managed to submit before he could.  However, the solution posted was incomplete, and users on mac later reported the issues still existed, and the issue reopened.  The developer from vscode attempted to make his own fix as well, but that did not work either.

To solve this issue, I continued where the walkthrough left off.  The first thing I did was to ensure the issue still existed, by quickly replicating the problem again.  I created an empty file with a space in the file name and attempted to open it through vscode.

This slideshow requires JavaScript.

From the screenshots above, we know the problem still existed.  The next thing I did was to explore what the developer had done recently to fix the problem.  It was posted in the comments, and his changes are here.  His fix was to remove all occurrence of white space from the file path using the following regex:
stripWildcards(original).replace(/\s/g, '');

If the space was removed in the file name, then quick open would not be able to open that file since it cannot match the file name.  My first thought to going around this problem was to look at how my fedora handles file names with spaces in the terminal.  I used the ‘tab’ function in the terminal to autocomplete my file name ‘temp Text’ and here are the results:

After typing ‘temp’ and then pressing tab, the terminal autocompletes the file name as ‘temp\ Text’

The other solution would be to have single quote (‘) or double quotes (“) around the file name in order to allow for spaces in the file name.  With this in mind, I decided to use this behaviour for pushing through the file names.

I changed the regular expression to the following at first:

stripWildcards(original).replace(/\s/g, '\ ');

However, this fix failed the test case of:

assert.equal(scorer.prepareQuery(' f*a ').value, 'fa');

The problem here was that if there were leading and trailing spaces in the filename, my regex would keep those spaces.  I needed something that would eliminate those spaces, without removing the ones in the middle of the string.  The next solution I had was the following:

stripWildcards(original).trim().replace(/\s/g, '\ ');

This trim() would eliminate the leading and trailing spaces without removing the whitespace in between.  The replace function would replace all the whitespace with an escaped space, similar to the behaviour in Fedora terminal.  I recompiled the vscode OSS version using the yarn run watch command, and then launched it with scripts/code.sh.  Afterwards, I typed in a full file path to a file name with a space, and vscode was able to find the file, as per the screenshots below.

This slideshow requires JavaScript.

From the screenshots, it would seem that my fix worked. The next thing I did was to run the test cases in quickOpenScorer.test.ts.  From running the tests, I failed the following two tests:

assert.equal(scorer.prepareQuery('model Tester.ts').value, 'modelTester.ts');

assert.equal(scorer.prepareQuery('Model Tester.ts').lowercase, 'modeltester.ts');

After much deliberation, I decided it was best to change the two test cases, as it does not reflect what the function is trying to do.  It was not a hasty decision to make, as I am sure there may be underlying reasons as to why the test cases were written this way to begin with.  However, based on the documentation above the function:

Helper function to prepare a search value for scoring in quick open by removing unwanted characters.

I concluded that the spaces in between the filenames were NOT unwanted characters, and that they should be kept so that files could be found.  For that reason, I updated the test cases to the following:

assert.equal(scorer.prepareQuery('model Tester.ts').value, 'model Tester.ts');

assert.equal(scorer.prepareQuery('Model Tester.ts').lowercase, 'model tester.ts');

In addition, I added more test cases to ensure the tests I made would be checked for breakage in the future.  In the end, the following is the updated test function:

	test('prepareSearchForScoring', function () {
	assert.equal(scorer.prepareQuery(' f*a ').value, 'fa');
	assert.equal(scorer.prepareQuery('model Tester.ts').value, 'model Tester.ts');
	assert.equal(scorer.prepareQuery('Model Tester.ts').lowercase, 'model tester.ts');
	assert.equal(scorer.prepareQuery('ModelTester.ts').containsPathSeparator, false);
	assert.equal(scorer.prepareQuery('Model' + nativeSep + 'Tester.ts').containsPathSeparator, true);
	assert.equal(scorer.prepareQuery('dir' + nativeSep + 'tester.ts').value, 'dir' + nativeSep + 'tester.ts');
	assert.equal(scorer.prepareQuery('dir' + nativeSep + 'model tester.ts').value, 'dir' + nativeSep + 'model tester.ts');
	});

view raw

quickOpenScorer.test.ts

hosted with ❤ by GitHub

The test cases test for file paths with their native separators and ensures the spaces in between the file names are retained.  Before I finished and pushed my changes onto my origin repo, I ran the test suite once more to ensure everything was running correctly.  Here are the results:

All test cases passed 🙂

With all the test cases passed, I proceeded to push to origin and submit the PR.  Here’s the link to the final results.

Issue 4:

The fourth issue is also from VSCode, and it also involved regular expressions with file paths.  This time it involved url paths and how VSCode adds a ‘/’ to the file path depending on the situation. The issue is linked here, and it has been there for awhile since last September 2017.  The reporter links the test cases that passed and failed in the initial post, and there are additional test cases added below the initial comment.

This bug has more to do with how different operating systems handle file paths rather than a broken functionality issue.  In it, the contributor and members of the community discussed back and forth about how strict and lax the regular expression should be.  One of the first things I did this time around was to read this conversation to ensure that there isn’t already a solution to this issue.  I also followed the additional test cases they were testing for to ensure my fix would pass those tests as well.

One of the tools I used to test my regex was the following: https://regex101.com/

It allowed me to test my regex to see the strings that are matched and the ones that are discarded.

Since the issue involved relative paths, and determining when to add the extra slashes, I decided to leave the logic for the windows platform cases intact and focus on non-Window test cases.  Instead, I added a regex:

const _isRelative = /^([~\.]\/|^[a-zA-Z]+\/)/;

This is to determine if we are using a relative path, based on the test cases provided.  It would look for either a ‘./’ , ‘~/’ , or any number of the alphabet followed by a ‘/’ to determine that the file path is relative.  Afterwards, we would only add a additional ‘/’ to the beginning of the filepath if it is NOT a relative path.

However, my original fix failed to the test case linked here.  The test case is for a non-Windows path with the string, ‘./c/win/path‘.  The expectation is that we do not add a ‘/’ to the beginning of this path.  After some exploration with the makeFsPath() function, I was able to pinpoint the spot to do an additional check for strings that start with the ‘.’, and ensure it does not append the extra slash.

Lastly, I added the additional tests to the suite as per follows:

	if (!isWindows) {
	test('absolute path', function () {
	const path = '/foo/bar';
	assert.equal(URI.file(path).path, path);
	});
	test('relative path', function () {
	const path = 'foo/bar';
	assert.equal(URI.file(path).path, path);
	const fileUri1 = URI.parse(`file:foo/bar`);
	assert.equal(fileUri1.path, 'foo/bar');
	const uri = fileUri1.toString();
	assert.equal(uri, 'file://foo/bar');
	const fileUri2 = URI.parse(uri);
	assert.equal(fileUri2.path, '/bar');
	assert.equal(fileUri2.authority, 'foo');
	});
	test('relative path with dot', function () {
	const path = './foo/bar';
	assert.equal(URI.file(path).path, path);
	});
	}

view raw

uri.test.ts

hosted with ❤ by GitHub

I ran the testers again to make sure everything in that suite is successfully passed prior to submitting the PR.

All test cases for uri.test.ts passing

This is my PR submission here.

Unfortunately, I made a mistake when pushing the commits by including my previous issue’s commits in this PR. I would like to find a way to remove those unnecessary commits if possible.

Learning experience and some thoughts:

By looking at the actual code in VSCode, and Brave browser-laptop, I would have to say the documentation in VSCode is more thorough and detailed.  I think this was one of the reasons why I could not follow Brave’s code once it got to filtering.js file.  The other reason I was not able to continue with the Brave issue was that I could not get the Brave debugger working for my Fedora machine.  It made me quite envious of how my prof was able to launch the Brave debugger from VSCode.

One of the issues that cost me a bit of time in VSCode was apparently formatting the code before being able to push the commits onto GitHub.  I tried many things such as running ‘gulp tslint’ which timed out after 46 seconds.  I also tried the tslint extension in VSCode, but that did not do the trick either.  Finally, I found the solution here:

Apparently, all I had to do was press “Ctrl + Shift + i” and it auto formats the file for me.

Lastly, thank you for reading this very long blog post.  I hope to fix more bugs in the future.

 

Brave URL Testing

On my latest lab for open source development, I was tasked to fix an error in the Brave laptop/desktop web browser involving parsing of URL from the address bar.  To fix this bug, I had to build the browser based on the source code from GitHub. Afterwards, I input several url test cases into the address bar and observe the behaviour from the Brave browser.  Finally, I repeat the test cases on the Firefox and Chrome browsers to see if there were any differences in how the url is handled.

From the lab, there were many test cases offered, but ones that were of concern are as follows:

• https://www.google.ca/search?q=dog cat
• /home/omak/Documents/dog cat.txt

For these test cases, the Brave browser will not treat the url as valid, and run a google search based on the url text.  For the screenshots below, I used the url: https://www.google.ca/search?q=dog cat and tested the browser responses for Brave and Firefox.  In Brave, the browser would perform a google search with the url input as keyword.  This is in contrast to Firefox, where the browser would parse the url and decipher the search terms to only be “dog cat”.

This slideshow requires JavaScript.

To get the class started on fixing the bug, our professor introduced us to the code involved in Url parsing for Brave and Firefox.  For Firefox, it was based on C++ code that went as far as the Netscape days, whereas for Brave, it was Javascript code.

Since this was a test driven development style of programming, I was supposed to write a set of test cases prior to changing the code in urlutil.js.  The first function I studied closely was the getUrlFromInput() function, because I felt that if I could change the way the string is retrieved from the address bar, I can control what is passed to the subsequent function calls.  To begin, I wrote a new test case for the getUrlFromInput as per the gist below:

	describe('getUrlFromInput', function () {
	it('returns empty string when input is null', function () {
	assert.equal(urlUtil.getUrlFromInput(null), '')
	})
	it('returns empty string when input is undefined', function () {
	assert.equal(urlUtil.getUrlFromInput(), '')
	})
	it('calls prependScheme', function () {
	assert.equal(urlUtil.getUrlFromInput('/file/path/to/file'), 'file:///file/path/to/file')
	})
	it('calls prependScheme with space in file name', function () {
	assert.equal(urlUtil.getUrlFromInput('/home/omak/Documents/dog cat.txt'), 'file:///home/omak/Documents/dog%20cat.txt')
	})
	})

view raw

urlutilTest.js

hosted with ❤ by GitHub

Strangely, when I ran the new test case with the expectation of failure, the following happened:

My new test case with file path to “dog cat.txt” passed.

Since this test case passed, my professor indicated to me that it meant getUrlFromInput() is not the function causing the erroneous behaviour.

I moved on to looking at another function in the urlutil.js of particular interest.  This time, it was the isNotURL() function.  By manipulating the inputs that get flagged as a valid urls, I can control the strings that processed as a Url and the strings that are processed as keyword searches. To start things off again, I wrote two test cases for the isNotURL() function, located at the bottom of the test cases:

	describe('isNotURL', function () {
	describe('returns false when input:', function () {
	it('is a valid URL', function () {
	assert.equal(urlUtil.isNotURL('brave.com'), false)
	})
	it('is an absolute file path without scheme', function () {
	assert.equal(urlUtil.isNotURL('/file/path/to/file'), false)
	})
	it('is an absolute file path with scheme', function () {
	assert.equal(urlUtil.isNotURL('file:///file/path/to/file'), false)
	})
	describe('for special pages', function () {
	it('is a data URI', function () {
	assert.equal(urlUtil.isNotURL('data:text/html,hi'), false)
	})
	it('is a view source URL', function () {
	assert.equal(urlUtil.isNotURL('view-source://url-here'), false)
	})
	it('is a mailto link', function () {
	assert.equal(urlUtil.isNotURL('mailto:brian@brave.com'), false)
	})
	it('is an about page', function () {
	assert.equal(urlUtil.isNotURL('about:preferences'), false)
	})
	it('is a chrome-extension page', function () {
	assert.equal(urlUtil.isNotURL('chrome-extension://fmfcbgogabcbclcofgocippekhfcmgfj/cast_sender.js'), false)
	})
	it('is a magnet URL', function () {
	assert.equal(urlUtil.isNotURL('chrome://gpu'), false)
	})
	it('is a chrome page', function () {
	assert.equal(urlUtil.isNotURL('magnet:?xt=urn:sha1:YNCKHTQCWBTRNJIV4WNAE52SJUQCZO5C'), false)
	})
	})
	it('contains a hostname and port number', function () {
	assert.equal(urlUtil.isNotURL('someBraveServer:8089'), false)
	})
	it('starts or ends with whitespace', function () {
	assert.equal(urlUtil.isNotURL(' http://brave.com '), false)
	assert.equal(urlUtil.isNotURL('\n\nhttp://brave.com\n\n'), false)
	assert.equal(urlUtil.isNotURL('\t\thttp://brave.com\t\t'), false)
	})
	it('is a URL which contains basic auth user/pass', function () {
	assert.equal(urlUtil.isNotURL('http://username:password@example.com'), false)
	})
	it('is localhost (case-insensitive)', function () {
	assert.equal(urlUtil.isNotURL('LoCaLhOsT'), false)
	})
	it('is a hostname (not a domain)', function () {
	assert.equal(urlUtil.isNotURL('http://computer001/phpMyAdmin'), false)
	})
	it('ends with period (input contains a forward slash and domain)', function () {
	assert.equal(urlUtil.isNotURL('brave.com/test/cc?_ri_=3vv-8-e.'), false)
	})
	it('is a string with whitespace but has schema', function () {
	assert.equal(urlUtil.isNotURL('https://wwww.brave.com/test space.jpg'), false)
	})
	it('has custom protocol', function () {
	assert.equal(urlUtil.isNotURL('brave://test'), false)
	})
	it('is a url which contains a space in the query', function () {
	assert.equal(urlUtil.isNotURL('https://www.google.ca/search?q=dog cat'), false)
	})
	it('is an absolute file path with a space in the files', function () {
	assert.equal(urlUtil.isNotURL('/home/omak/Documents/dog cat.txt'), false)
	})
	})

view raw

urlutilTest.js

hosted with ❤ by GitHub

The test cases failed as expected, and I knew from this failure that I had to modify some of the code in the isNotUrl() function.

To fix the behaviour in Brave, I had to work with 3 cases.  The first case is to check if there was a scheme to the url, and then look for a space in the url text.  If there is a scheme, then the url is definitely not a file path, and what I needed to do was to replace the whitespace ONLY in the query section of the url string with ‘%20’.  At first, I simply replaced all whitespace in the entire url string and triggered a new error in one of the existing test case:

assert.equal(urlUtil.isNotURL('https://www.bra ve.com/test space.jpg'), true)

My fix would mistakenly trigger an error on this case because it would replace the whitespace in ‘bra ve’ and change it to ‘bra%20’.  Afterwards, the browser would attempt to go to url https://www.bra%20ve.com/test space.jpg and result in an error.

The function mistakenly adds the ‘%20’ when there is a space in the domain name

To prevent this error, I had to check the case of an existing scheme (http or https), and then check for a space in the url string.  Afterwards, I would search for a ‘?’, such that I will only replace the whitespace in the query part of the url.  Here’s a short snippet of the code I used to perform this replacement:

	if (/\s/g.test(str) && scheme !== undefined) {
	// attempt to change query whitespace to %20
	var queryIndex = str.search(/\?.*$/)
	if (queryIndex > 0) {
	var query = str.substring(queryIndex).replace(/\s/g, '%20')
	str = str.substring(0, queryIndex) + query
	}
	}

view raw

urlutil.js

hosted with ❤ by GitHub

After the small addition, passing an url such as https://www.google.ca/search?q=dog cat hamster would replace only the spaces in the query string, and leave string in domain name unchanged.  The following slides demonstrates this behaviour.

This slideshow requires JavaScript.

The last two cases of ‘dog cat’ and ‘/home/omak/Documents/dog cat.txt’ would be fixed under the same section of the isNotUrl() function.

	if (case2Reg.test(str) || !case3Reg.test(str) ||
	(scheme === undefined && /\s/g.test(str))) {
	if (scheme === undefined && /\s/g.test(str)) {
	// check to see if undefined scheme is possibly file scheme
	let str1 = UrlUtil.prependScheme(str)
	if (UrlUtil.isFileScheme(str1)) {
	// if it is file scheme, will then replace whitespace with %20
	str = str.replace(' ', '%20')
	} else {
	return true
	}
	} else {
	return true
	}
	}

view raw

urlutil.js

hosted with ❤ by GitHub

This code will only replace the whitespace if the url is a file path.  This means only file paths such as ‘/home/omak/Documents/dog cat.txt’ will be converted to ‘/home/omak/Documents/dog%20cat.txt.  Strings such as ‘dog cat’ will remain as google searches and left untouched.

This slideshow requires JavaScript.

With the two fixes, I managed to get Brave to work for the following urls:

• https://www.google.ca/search?q=dog cat
• dog cat
• /home/omak/Documents/dog cat.txt
• https://www.bra ve.com/test space.jpg

This is a final screenshot to show my test cases are working at the end.

Take particular note in ‘is an absolute file path with a space in the files’ and ‘is a url which contains a space in the query’

The commit to this fix is located here.

Learning experience:

From this exercise, the biggest learning experience was from comparing the way firefox wrote their code in C++ to the way it was written in Javascript for Brave.  By comparing the two codebase, one could tell a lot has changed in software development from the days of Netscape to current coding practices.  In particular, in the Firefox code base, the naming of variables based on facts about the variable, such as it being a member variable, therefore they put a ‘m’ before the variable name was quite interesting for me.  Another interesting note was the canParseURL() function in the urlutil.js, where the professor explained the case of window being undefined, highlighting the possibility of the function not being used from a browser.

This exercise was also my first time trying Test Driven Development, and it was quite different for me to write out my test cases first, and then modify the code base.  As mentioned above, I was quite surprised to have my test case on getUrlFromInput() pass, when it should have failed.  This scenario demonstrated the merits of writing out the test case first, as I would have wasted a lot of time fiddling with the getUrlFromInput() method otherwise.

 

Trolls and Sunshine

For the latest lab in my Open Source Development course, our professor designed a bridge troll project which maps bridge data in Ontario.  The premise is that there are trolls hiding beneath bridges, and the app will use geolocation to find out your current position, and mark the bridges in your surroundings for possible troll sightings.

In order to run the app to test things out, I had to fork the repo into my Github account, and then clone to my local machine.  However, as the upstream repo got more and more updated, I found myself having to make pulls to the upstream, and then push those updates into my origin.  This made me wonder if there was another way to keep the forked repo in my own account automatically updated to keep in sync with the upstream repo.  Nonetheless, since I am still learning to use Git properly/intuitively, I found myself having to start fresh several times, wiping up my local repo and re-cloning from either upstream or origin.

Frameworks used by bridge-troll:

After downloading the repo, I had to use npm to build the app.  This was very smooth and painless.  Some of the packages the project used include: babel, parcel and prettier.  Babel is a transpiler which can take javascript code based on the latest standards, and transform the code into browser compatible javascript.  This allows programmers to code using the latest javascript without having to worry about browsers not being able to render it.  Our prof compared it to the Tower of Babel mythology that was used to explain why we have so many spoken languages today.

Parcel is a bundler that groups assets like CSS, images, html and javascript files together and sorts out the dependencies.  This allows for easier importing around the code, translates the require() statements URLs so that the browser can render it.  The main feature for Parcel over Webpack is that Parcel does not require configurations, because it supports CSS, HTML, file assets and JS out of the box.  While trying to understand what a bundler was, I found this blog which explained the purpose of bundlers like webpack.

Another module the project used was Prettier.  This allowed the code to have consistent formatting by automatically reformatting the code to follow their style.  The modules allows a project with many contributors to end up with code that follows a consistent style.

The task at hand:

One of the feature request for this project was to allow for changing of themes for the map that is displayed.  The goal was to use a darker palette for the maps during night time and to use a lighter, more colourful palette during daytime instead.  To achieve this goal, we need 3 things:

1. Calculate the times for sunrise and sunset
2. Changing tile sets used in the map
3. Changing the icons used in the map

To calculate the sunrise and sunset times, I had imported an additional module into the project called sunCalc.  This module calculates all relevant times such as sunrise, sunset, dawn, night and golden hours when provided a latitude and longitude.  Since we are concerned with changing to a darker palette when the sun goes down, the suncalc times that are relevant are the sunset and sunrise hours.  We would want to change to a darker theme after the sun goes down, and before the sun rises.  Below is a screenshot of the code I wrote to determine when to switch into a dark theme:

The next thing is to figure out how to change the tile sets.  From looking at the code, I realized the tile sets are changed in base-ui.js (formerly map.js).  From the description in the feature request, I also know that the project uses leaflet to import tile sets used in the mapping.  All I had to really do is browse through a collection of tile sets here, and select the tile sets I would like to use.  Afterwards, I used the manageViewMode module I coded to get the mode I should be displaying, and feeding the corresponding url for the tile set I would like to display into a variable called tileUrl.  Below is the relevant code I used to change the tile set in base-ui.js:

Lastly, I would like to change the icons into the white coloured version during nightime.  The only icons I changed were the locked, unlocked, and location icons.  Icons such as the map icon were left unchanged.  To do this, I searched and downloaded each respective icon from Material Icons and added the icons to the icons folder of the project.  Afterwards, I modified each icon’s file path such that it was dependent on the display mode in svg-marker.js file.

Searching for the white version of the location icon.

After making these changes, I uploaded my fix onto my GitHub repo, and then submitted a pull request into the bridge-troll project.

Here are some screenshots of the final look:

The map during day mode.

Night mode for the map. Notice how the icons are now white. This was the only tile set I found which could clearly display the white icons.

Overall experience:

The majority of the difficulties from implementing this feature came from my lack of javascript knowledge.  I have not done much work in javascript since over a year ago where I had an introductory course into js, css, and html.  My lack of proficiency in js not only made it very difficult to understand the code used in the project, but I found myself slowly reacquainting to basic js syntax as I was coding the new features.  This lengthened process of implementing the feature by much longer than necessary.

Another difficulty I had was from the windows environment.  Even though Git changes the line endings for the code during checkouts, the eslint was giving me issues when coding in windows.  As a bypass, I had to switch to my Fedora laptop to do most of the coding instead.

The last difficulty of note is the testing of this application.  At the beginning of the project, I had not know much about conditional debugging, and the use of log levels to reveal the testing statements made.  As a result, I spent a lot more time guess and checking to see which statement was breaking the app.  After being shown how to use log.info('Some useful test statement here'), I was able to display my test statements passing the parameter ?loglevel=info in the url.  This revealed important facts such as the current time or the sunrise times during the execution of the app, which simplified the debugging process for me.

The pull request I submitted can be found here.

VSCode – Titlebar bug

For my second release, I finally settled down working on a VSCode bug regarding the title on the title bar of the VSCode window.  The issue can be found here within the issues tab of the VSCode Github repo.

So what is the bug?

In VSCode, the user can change the settings of the editor.  VSCode is shipped with a set of default settings, and the user can override the default settings with their own user settings or workplace settings.  The settings are also grouped under categories such as Editor, Workbench, Window and Source Code Manager.  The settings menu can be accessed under File > Preference > Settings.  Below is a screenshot of the settings menu illustrating the default settings and user settings:

At the center column, we have the default settings, and at the right column, we have the User settings.

The bug involved changing the “window.title” setting from:

"window.title": "${dirty}${activeEditorShort}${separator}${rootName}${separator}${appName}"

to:

"window.title": "${activeEditorLong}"

The “window.title” settings controls the title displayed at the title bar of the VSCode window.  For example, if you have the following:

Variable used	Displayed title
${activeEditorShort}	myFile.txt
${activeEditorMedium}	myFolder/myFile.txt
${activeEditorLong}	/Users/Development/myProject/myFolder/myFile.txt

In other words, the “window.title” controls the format of the title displayed at the title bar.  The setting we are concerned with for this bug is the ${activeEditorLong} setting.

When we use the ${activeEditorLong} setting, the full path to the focused file should be displayed at the title bar.  In the screenshot below, the focused file is command.ts, and the title bar displays the full path to command.ts.

The bug occurs when we use the “Open Changes” button at the top right of the VSCode window.  The open changes button creates a new tab and allows the user to compare between the saved version of the code to the last committed version of the code in the git repository (assumes we use Git as the SCM).  However, when we use the “open changes”, the title at the title bar no longer displays the full path, but instead uses a short path.

This slideshow requires JavaScript.

This is the basic summary to the bug, and next we will discuss the steps I went through to fix it.

How was the bug fixed?

From the start, I was given the tip to look at command.ts for implementing the fix.  One of the key members of the project had pointed out to look into command.ts in one of the comments.  The first thing I had to do was to figure out how to change the settings to "window.title" : "{activeEditorLong} and how to use the “Open changes” button.  Afterwards, I started playing around with the code to see if I could change the title, by looking into the path definitions, the Resource definition and the functions available for resourceUri.

This slideshow requires JavaScript.

I was able to modify the title at the title bar from the command.ts > getTitle() function using functions such as path.basename() and path.dirname().  The problem with the changes implemented in command.ts was that it affected all settings of “window.title”.  This meant that any changes made in command.ts not only applied to  "window.title" : "{activeEditorLong}, but  "window.title" : "{activeEditorMedium} and all other settings for “window.title” as well.  To solve the issue, I would have to find out where the settings are verified and processed, and apply the changes at that point.

To find out the locations of where the “window.title” settings were used, I performed a search on Github to get some clues using one of the settings, ${activeEditorMedium} as the keyword.  The results of the search were as follows:

I had to filter out JSON and view only Typescript related files

From the results of the search for keyword: activeEditorMedium, it was apparent that this setting was also called in the file: titlebarPart.ts.  Of particular interest was the doGetWindowTitle() within this module as it’s comments indicated all of the template values.  From this function, we can immediately tell that the “window.title” is retrieved with  const input = this.editorService.getActiveEditorInput(); at line 186, and that the input is checked and compared to later in the function.  This means we can assert from this section of code that the setting is ${activeEditorLong} before making changes to the title at the title bar.  In addition, we can check for the operating system in order to use the correct separator for file paths (ie, ‘/’ vs ‘\’). Lastly, we can change from an absolute path to a relative path for linux systems as per the first post in the issues.

This is a screenshot of the changes made in titlebarPart.ts:

I checked for the operating system and the settings first, and then I had a regular expression to check for absolute paths in the template.  Lastly, I figured out to return the entire file path, I can combine ${folderPath} variable with the ${activeEditorMedium} variable to form the full path.  Here are some of the results from the changes:

This slideshow requires JavaScript.

Here’s a link to the PR: https://github.com/Microsoft/vscode/pull/46466

What I learned from this experience

I had made several errors while working on this project.  The most critical one being taking too long before I settled on a project.  I was given ample time to find an issue and fix it, but I spent too long browsing between projects before settling on one.  One of the reasons I took so long was the intimidation from browsing individual issues.  The technical aspects to understanding the root causes of the issues posted thwarted me from committing to the project early on.  As time ran out, and the deadline for the assignment loomed, I had to finally muster the courage and just pick one bug attempt fixing it, instead of avoiding the bug to browse some other project.

A second thing I learned is that the cause of the issue may not always be the same place indicated by the members of the project.  In this case, I was pointed to command.ts by comments from a regular contributor when I worked the issue early on.  After hours of tinkering with the code, I had realized that to fix the title for certain settings, I had to go to titlebarPart.ts instead.

The last thing I learned from this experience was the need to communicate ahead of time to ensure the issue I was working on actually required a fix.  In this case, after submitting the PR, my PR was turned down because they apparently wanted a new concept for setting the title instead.

Merry Go around – quest for finding a project to contribute

In the last several weeks, I was tasked with finding an open source project for my class and submitting a pull request to fix an issue.  I looked at several project such as Elasticsearch, the Brave browser (both desktop and mobile version), Teammates and lastly VSCode.

The biggest problem I had with this assignment was settling on a project and finding a bug to fix.  It was like window shopping projects without making the commitment to contribute.  At first, I tried to narrow down my project search to projects using Java, since I would likely be programming with Java in my upcoming coop work term.  However, there are not too many major projects on Github which uses Java, and this limited the scope of my search.  One of the resources I came upon to help find a project is a Github repo containing many links to projects with “Good First PRs”.  From here, I tried out Elasticsearch and TEAMMATES.

Elasticsearch:

The first project I looked at is Elasticsearch. It is an open source search engine built using RESTful concepts.  It can perform text searches using GET requests and indexing searches using POST or PUT requests.  Besides using RESTful API, it also has APIs for other programming languages such as Java, Python and Javascript.  Their Gitub page can be found here and a list of languages it supports can be found here.  After building the project from the cloned repo, I played around with indexing a resource and searching for that resource.  Eventually, I used postman to create HTTP messages for searching and indexing instead.  Below are some of the screenshots I took as I was testing out the software.

This slideshow requires JavaScript.

After testing out Elasticsearch, I successfully setup the development environment and was ready to look for issues I could contribute to. However, I came upon a section in the CONTRIBUTING.md which stated that the project did not want short term contributors who were making PRs as part of class.  As a result, I moved onto browsing for another project to work on.

Teammates:

The second project I looked at was TEAMMATES.  It is an educational software for instructors to manage their courses sending evaluations / feedback for the students enrolled in the course.  The web application is written in Java and runs on google app engine.  The one thing of note about this project was the very detailed, step-by-step guide to not only run the project, but to setup the IDE as well.  From setting up google cloud platform in eclipse, to setting up the indentations the IDE is using, the project goes into detail to make it easy for newcomers to get on board.

Brave browser – laptop/desktop:

Another project I looked at is the Brave browser.  This is an open source project to create a secure web browser that comes with a built-in blocker for ads and trackers.  This takes away the need to install 3rd party adblockers when using the browser.  From my experience using the desktop version of the browser, the first noticeable feature was previewing a tab by hovering over it.  The browser would display a dimmed version of the tab being hovered over.  The desktop version of the source code was written mostly in javascript, while the mobile version of the code was done in java.  The reason I did not end up working on this project was because I couldn’t setup the developer environment in my windows desktop to build the project.  The build error I got was as follows:

Another project I looked at is the Brave browser.  This is an open source project to create a secure web browser that comes with a built-in blocker for ads and trackers.  This takes away the need to install 3rd party adblockers when using the browser.  From my experience using the desktop version of the browser, the first noticeable feature was previewing a tab by hovering over it.  The browser would display a dimmed version of the tab being hovered over.  The desktop version of the source code was written mostly in javascript, while the mobile version of the code was done in java.  The reason I did not end up working on this project was because I couldn’t setup the developer environment in my windows desktop to build the project.  The build error I got was as follows:

Another project I looked at is the Brave browser.  This is an open source project to create a secure web browser that comes with a built-in blocker for ads and trackers.  This takes away the need to install 3rd party adblockers when using the browser.  From my experience using the desktop version of the browser, the first noticeable feature was previewing a tab by hovering over it.  The browser would display a dimmed version of the tab being hovered over.  The desktop version of the source code was written mostly in javascript, while the mobile version of the code was done in java.  The reason I did not end up working on this project was because I couldn’t setup the developer environment in my windows desktop to build the project.  The build error I got was as follows:

Brave browser-laptop:

Another project I looked at is the Brave browser.  This is an open source project to create a secure web browser that comes with a built-in blocker for ads and trackers.  This takes away the need to install 3rd party adblockers when using the browser.  From my experience using the desktop version of the browser, the first noticeable feature was previewing a tab by hovering over it.  The browser would display a dimmed version of the tab being hovered over.  The desktop version of the source code was written mostly in javascript, while the mobile version of the code was done in java.  The reason I did not end up working on this project was because I couldn’t setup the developer environment in my windows desktop to build the project.  The build error I got was as follows:

After spending hours trying to overcome this error, I decided to move on and browse for another project instead.  Note: I was able to build this repo on my linux laptop, but not on the Windows desktop computer.

VSCode:

 

VSCode is the project I returned upon after the merry-go-around hunt for bug fixing.  I already had the environment setup from previous labs in my course work, so it was only a matter of finding a bug that I could fix.  VSCode is an open source editor with built-in Git support, and extensions support from 3rd parties.  This allows contributors to implement new features and customization for the editor such as linting, developer tools / debugging, and user interface themes.  The source code for VSCode can be found here, and a running version of the software can be downloaded here.

VSCode also provides documentation on how to contribute which can be viewed here.  To get involved in the project, simply follow the instructions in the how to contribute guide to setup the environment.  Afterwards, look through the issues posted on Github and see if you could replicate the bug.  Once you have fixed the bug, you can push the fix to the origin repo, and finally make a pull request to the upstream repo.

Interesting things from browsing for projects:

The first interesting I found was when I looked over the issues in Elasticsearch and came upon this.  The issue involved open source licensing for one of the dependencies in the project, org.json.  The core of the issues lies on an additional clause: “This software shall be used for Good, not Evil.” in the MIT license of org.json.  Apparently, this additional clause created a furor back in 2009 resulting in Google no longer recognizing the customized license as “free” and “open” software.  Today, most open source organizations such as Apache, Fedora, Red Hat no longer view this custom license as open and free as well.  More information about this issue can be found here.

Another interesting note I came upon was when I browsed through the TEAMMATES project.  From the project’s Github README.md, I was redirected to a competition called “Google Summer of Code application“.   The idea was to come up with a new project/feature to be implemented on Teammates, and draft a proposal for this project.  After an approval process, the selected candidate will work under the mentor organization for the summer to implement this project.

 

Side Quest #2 – ECMA and Unit Tests

Today, my Open Source Development course talked about Open Standards and Open Data, from how it used to work to how it currently works.  From my previous course in Data Communications Fundamentals, I learned about organizations such as the ITU-T, which is an agency within United Nations responsible for maintaining telecommunications standards like VoIP.

In C++, the standards are maintained under the ISO (International Standards Organization).  However, in javascript, standards were developed and updated by ECMA, European Computer Manufacturers Association.  ECMAScript Language Specification, also known as standard ECMA-262, is a standard for javascript first released in 1997 and subsequently updated as the web evolved.  Today, the latest standard is the ECMA-262 8th Edition released in 2017, with added features such as async.

Within the standard comes specifications and unit tests that ensure the build behaves up to those specifications.  During class, we looked at the unit tests for the UpperCase() function in javascript.  We discussed how the testing framework has changed with functions such as $ERROR becoming deprecated, and new unit tests using the Assert function instead.

For my task today, I am to first build the test suite that ECMA used to test their standards, and then write a few test cases for the reverse() method.  To build the test suite, I cloned the github repository from here, and then ran the command: test262-harness test/**/*.js.

This command ran all 205 tests within the test suite and resulted in the following:

Six of the tests failed, most of which are about locale.  In particular, it was the supportedLocalesOf-default-locale-and-zxx-locale.js and the default-locale-is-supported.js that failed.

After running the tests, I had to read the Ecma specifications for the reverse() function in arrays.  From what I could tell from the specs, the reverse() function has to return an object type, and that it uses the numerical indices to swap the first half with the 2nd half of the array .  In particular, it iterates from index 0 to the midpoint, calculated by floor(len/2), and flips the upper and lower parts of the array.

Once I finished reading the specs, I started looking at the unit tests code for the reverse() function in Array and TypedArray.  It was hinted that there were developments made when writing the unit test cases in TypedArray’s reverse function which could be applied to Array as well.

The first interesting thing that caught my eye was a new function called compareArray() used in revert.js of the TypedArray tests.  Instead of comparing the array one element at a time, the compareArray() takes two arrays, compares it, and returns a boolean value based on the results.  I was curious where this function came from, and used the shortcut ‘T’ in github to search for this function:

After reading the compareArray() code, I decided to call this function later on when I write my test cases.

Another thing I noticed when browsing through the TypedArrays tests was a file named “preserves-non-numeric-properties.js“.  This unit test checks for non-numerical keys, and ensures the key value pair is not affected by the reverse function.  For example, if TypedArray A = [1, 2, 3, 4, foo: “foo”], the A.reverse() should be equal to [4, 3, 2, 1, foo: “foo”].  This means the values with non-numerical indices remain at the end of the array.

With both of these things in mind, I wrote my test cases and tested them locally first.  Afterwards, I uploaded the cases in GitHubGist.  The following are my test cases:

	// Copyright 2018 Owen Mak. All rights reserved.
	// This code is governed by the BSD license found in the LICENSE file.
	/*—
	info: |
	The elements of the array are rearranged so as to reverse their order.
	The object is returned as the result of the call
	esid: sec-array.prototype.reverse
	es6id: 22.1.3.20_A1_T2
	description: Checking this algorithm, elements are objects and primitives
	…
	includes: [testTypedArray.js, compareArray.js]
	—*/
	var x = [1,2,3,4];
	var y = [4,3,2,1];
	var reverse = x.reverse();
	//Check #1
	//ensure return type of array after reverse remains as object
	assert.sameValue(typeof(reverse), "object", "typeof(x.reverse()) === 'object'");
	//Check #2
	//returns same object
	assert.sameValue(reverse, x, "returns the same object");
	//Check #3
	assert.sameValue(reverse[0], 4, "reverse[0] === 4");
	assert.sameValue(reverse[1], 3, "reverse[1] === 3");
	assert.sameValue(reverse[2], 2, "reverse[2] === 2");
	assert.sameValue(reverse[3], 1, "reverse[3] === 1");
	//Check #4
	//using compareArray function from typedArray to compare the arrays
	assert(compareArray(reverse, [4, 3, 2, 1]));
	assert(compareArray(reverse, y));
	//Check#5
	//check for non-numeric values
	var x = [1,2,3,4];
	x.foo = "foo";
	y.foo = "foo";
	var reverse = x.reverse();
	assert.sameValue(reverse.foo, "foo", "reverse.foo === 'foo'");
	assert (compareArray(reverse, y));
	//Check #6
	//Check for string lengths
	assert.sameValue(x.length, 4, "length is 4");
	assert.sameValue(x.length, reverse.length, "length is preserved after reverse");

view raw

S22.1.3.20_A1_T1.js

hosted with ❤ by GitHub

The test cases are hosted here and they use the new features I mentioned earlier in this post.  From using assert instead of $Error, to using the compareArray() and testing for non-numerical values, I wrote my first set of test cases for the reverse function.

 

Solo Boss Fight: Interviews

I recently attended a few interviews for coop positions and were asked some thought provoking technical questions, so I decided to blog about this before I start forgetting.  The interview was for a Software Developer position in a well established financial tech company, and I had two interviews with them in total.

The first interview was a written tests where the candidates have an hour to complete several pages of coding and concept questions.  From the cover page which said “You are expected to answer most questions”, I knew it might not be possible for me to finish the test in the limited time frame, but the test was maybe to check how much I can do in that period instead.

The test started off with a programming question where I was given a scenario, and asked to write some code for it.  The question focused on arrays, where I was given three integer arrays of the identical size and I had to find the number of integers that appeared in more than one array.  To make the question easier, it was stipulated that none of the arrays contain duplicate elements, so I didn’t have to run array self checks.

Since there were three arrays of equal size, I only had to manage one array length when I started to code my loops.  Upon writing the header of my first loop, I realized it would be simpler to write a helper function, which I could use to compare two arrays, and return the number of elements that appeared in both arrays.  As a result, I quickly scratched off the header to my loop, and wrote the helper function instead.  I had to call the helper function three times in total, because the number of combinations between three arrays is 3.  In particular, if I had three arrays: A, B, C; then I had to run the helper function for the combinations AB, AC and BC.  Afterwards, I only needed to sum up the return values of the helper functions to get the total.

Besides the programming question, there were a few concept questions with short answers.  The ones that come to mind were:

1. Explain the advantages and disadvantages to manual testing versus automated testing
2. What are some of the different uses between client side programming versus server side programming?
3. What is the height of a binary tree?
4. Name the advantages of using the following languages: C++, Java, Python

The second programming question involved writing a function that takes a string such as “Battle of Helm’s Deep” and reversing the order of the words. The function would return “Deep Helm’s of Battle” in this case. To further test my knowledge, it was stipulated I could not use functions such as split(), reverse(), or Collections Frameworks from high level programming languages to build my solution.

My approach to this question was to break the string into an array of words as the initial step. I would have to assume the string is space delimited as shown in their example. However, since I could not use the Collections framework, I realized my choice to store the list of words would have to be a string array. To figure the number of words in the string, I had to count the number of spaces in the string, and add it by 1. That is, if a string had 1 space, then it would have two words. I decided it would be simplest if I wrote a helper function that counted the number of words in the string in this case.

After counting the number of words, I can then create the string array as I now knew the size of the array. With the string array built, I wrote a loop that kept track of the index position of the last character on the previous word, and iterated in search of space characters. Once a space was reached, it would take the substring from the end of last word, to the current index, which is the space, and treat it as a new word. The new word would then be added to the string array. Lastly, I would then update the index position of the previous word. Once the loop is completed, I had to take one of substring between the index of previous word, to the end the string as my last word. This is done since there is no space character after the last word.

The last step to the solution was to take the string array and reverse all of its elements. To do this, I made another string array of equal size, and simply populated the new string in reverse order using for loop.

The written tests had 2 more programming questions, and a few more concept questions, all of which I cannot remember. Overall, I was not able to finish all of the questions in the time frame, and had three questions unanswered in total. As I walked out of the test room, I wondered how many questions the other candidates completed.

… To be continued later when i get more time

Side Quest: Fixing bugs in VSCode

For this side quest, I had a selection of bugs to choose from for an attempt to fix it.  The bug I chose had to deal with {} characters while searching in a folders.  I chose this over the other bugs because I was able to replicate this error, and felt I had a good chance to tackle this bug.  The other bug I looked at had to do with markdown reference not linking properly when there was a forward slash escape sequence ( \ ).  I was not able to replicate this error in my VSCode, so I settled on the folder searching bug.

The {} character bug was originally tested under OS Version: Mac OS X El Capitan 10.11.6, but I was able to replicate it under my Windows 7 Home Premium 64-bit (6.1, Build 7601) machine.

What is this bug?

This bug is caused by having {} characters as part of a directory name.  To test this bug, I created two directories named hello and {hello}. Within the directories, I had a text file named “hello.txt” and “hello2.txt” which contained the words “hello”. Below is a screenshot of the file system I used to test the bug:

To test the bug, my first test case involved only searching for “hello” inside any file. As you can see in the screenshot below, a basic hello search will find both files with the “hello” text.

My next test is another hello search, but with an additional parameter of hello at the files to include.  This test results eliminates ./{hello}/hello.txt from the previous results and we are only left with ./hello/hello2.txt.

My last test is a hello search with {hello} at the files to include field. Logically, the expected search result is one file, with ./{hello}/hello.txt being found.  However, due to the bug, ./hello/hello2.txt is found instead.

Bug is found, but what is next?

From the last screenshot above, I surmised that under the files to include field, only hello is being passed between the method calls instead of {hello}.  To test this premise, the first thing I did was open up the Developer Tools.  The keyboard shortcut is Ctrl + Shift + I.  With the Developer Tools opened, I can then track the elements being modified when a search is initiated.

From the tools, I can tell which elements are modified when I perform a search. The tool also grants access to the debugger, allowing for break point insertions into the code itself.

From class, the professor searched for the keyword “ripgrep” on the VSCode library on GitHub, and found a file of interest called ripgrepsearch.ts.  However, I also noticed a file called search.ts within the same directory that appeared of interest to me, and decided to look into that file instead.

Without not much to go on, I inserted a break point to one of the first service calls inside search.ts, and decided to watch over an attribute called “includePattern”.  The goal was to see when this attribute changed from {hello} to hello.  From the screenshot below, it shows that the includePattern starts off perfectly fine as {hello}.

One of the things I noticed was that VSCode was treating { and } characters as word separators.  I am not sure if this will have an impact on the includePattern, but this seemed like a probable cause for the bug as well.

To narrow down the source of the bug, I stepped from my first breakpoint at line 273 of searchService.ts, to the statement that populated the faulty search results.  This narrowed the range of my bug hunt.  Below is a screenshot of the code the updates the search results onto the screen:

I now have a specific range to search for the bug.  Unfortunately, after hours of searching, I could only narrow the search between the line of code at searchModel.ts and the update result from earlier.  Below is the furthest point I managed to narrow down before I could no longer make sense of the code.

As highlighted, the includePattern is still {hello} at that point, and the query seems to be correct at that point.  After this point, the code jumped around to notifySuccess() calls, before reaching the update search results call.

Conclusion:

This was a good experience for me in bug tracking.  It exposed me not only to the Developer Tools built into VSCode, but I also used Github’s search functions to help narrow the scope of my searches.  I was confident that by using the debugger to step line by line, I’d eventually find the code which caused the bug, but it is apparent that there is more to bug tracking than that.

Hopefully in the future, as my debugging knowledge improves, I’d be able to go over hurdles such as the one I encounter today.

Mid Boss: Servlet deployment

My latest adventure into open source led me to create my first very own micro web service using google’s libphonenumber repository.  The goal was to create an open source API that can process Http requests and respond by listing all valid phone numbers in JSON format.  The specifications to the API is posted here, and the source code for my project is hosted here on Git hub.

To solve the problem posted by the specs above, I created a Java servlet called ParsePNum. It is a java class that extends the HttpServlet class, and its body contains  doGet() and doPost methods which handles the GET and POST requests mapped to a specific URL path.  As for testing, I used the JUnit framework and mockito to mockup servlet requests for tests.  Lastly, I created a jsp (Java servlet page) file that acted as the user presentation to my API.  Here’s a screenshot of the jsp file:

The textfield under GET accepts Canadian phone number(s), and the file upload under POST will find all phone numbers inside a text file.  The API uses the findnumber() method in google’s libphonenumber library to search for phone numbers.  As such, it will only work on phone numbers delimited by spaces, and does not work for stings such as “4162974913abcd4162974914”.

This assignment itself was intimidating at first, as I was overwhelmed not only the task to be accomplished, but by the number of possible ways to complete the task.  The google library supported many languages and the first step was to decide which language to build the API upon.  I browsed the phone number libraries of PHP, Java and C#, as they were the languages I was familiar with.  The amount of choices paralyzed me for the first two days leaving me out of my comfort zone as I did not  sense the path needed to build the API.  Fortunately, google made a few demos for their library to show users how to use it.  After testing and reading the code for the Java version of the demo, I settled on using Java for the API and developed a rough idea of what needed to be done.

Having decided to use Java’s servlet framework to complete the assignment, I had to decide upon the software tools needed to build the servlet.  I caught another lucky break when a fellow classmate, Patrick Godbout, linked me a tutorial with a step by step guide to create a basic servlet.  To build the servlet, I needed the Java EE version of Eclipse, and an Apache Tomcat server.  With both of these tools, I was able to code and launch my servlet locally.  With source code from google’s demo, I was able to create the skeleton version of my servlet, which met the requirements from the specs.

To write my test cases, I settled on using Mockito and JUnit to mock up HttpServlet requests and responses.  With mockito, I was able to write test cases for the doGet() of the servlet without problems.  However, I ran into trouble for the doPost(), as I could not fake a request containing a file upload.  I left testing the doPost() as an issue for other contributors, who were later able to resolve the issues for me using the servlet’s FileUpload class.

During class, my professor mentioned a software tool called Postman briefly, and indicated this may be a good tool for testing the API.  With Postman, I can customize a HTTP request, and send it to a specified URL without the use of a browser.  Additionally, I can write a collection of test cases that tests the response of web pages after it processes the requests.  Once again, I installed Postman, tried out the tutorial, and was soon writing test cases for Patrick’s version of the API as one of the two issues I had to work on.  Below is a screenshot of the requests I customized and along with the test cases used to test Patrick’s API:

The collection of tests were then exported as a JSON file and added to the upstream repository as a pull request.

For the second issue I had to work on, I decided to deploy Patrick’s API using Heroku onto the web.  Heroku is a cloud application platform which can launch web applications onto their servers.  To get a basic idea on how this worked, I followed this tutorial and launched a basic application locally at first.  Here’s a screenshot of the “hello world” app for heroku:

After successfully deploying the “hello world” application onto the web, I worked towards deploying a servlet instead.  Fortunately, there was another detailed guide written for launching servlet instead.  With the help of the guide, I was able to deploy Patrick’s API onto the web. Below is a screenshot of the finished deployment:

The name peaceful-fortress-77561 was randomly assigned by heroku, and the web application is under herokuapp.com’s domain.  In the near future, I would like to deploy my own web application using heroku as well.

Overall, this assignment was very enjoyable for myself, as it exposed me to new software tools such as Postman and Heroku.  In addition, I gained some experience using Travis CI as well, since a contributor helped setup my API for Travis.  For subsequent releases, I would like to learn more about OCR (Optical Character Recognition), and apply it to my API.

First Encounter: Building vscode

Today, I had my first encounter with building open source software from Github.  The task was to clone the latest version of Visual Code by Microsoft and try to rebuild this software from its source code.

Visual code is an open source code editor that aims to allow developers to build web applications.  The editor has its own marketplace for extensions which allows the editor to support a myriad of languages.  Some of the extensions I installed right away were the “Java Extension Pack”, “Vim” and “Debugger for Firefox”.  I’ve been using Java since last semester, so I thought it’d nice to run some of my existing code inside vscode.  Since the start of my studies at Seneca, I’ve also been using vim to do a lot of my coding as well.

If you would like to try out building your own vscode, the repository for the vscode project can be found here.

My laptop is currently runs on Fedora 25, which is a Linux distribution, so the instructions below are only applicable for Fedora platforms.  All of the commands below are run inside the vscode folder which you get after you clone the repository mentioned above. To build vscode from its source, there are 4 other software you need to setup first:

1. 1. 1. Yarn
         Yarn is a package builder that caches the dependencies you used to build your project. It can easily be installed with:
         sudo dnf install yarn
         For other systems, you can use go here for instructions on how to get yarn running on your machine.
      2. Node.js
         Node.js was slightly more difficult as I already had an old version of node on my machine.  This meant I had to upgrade my mode from version 6.1.4 to 9.4.0 using the command:
         sudo dnf -y install nodejs --best --allowerasing
         I used the guide here to help me install node.js.
      3. Python
         To install python, I used:
         sudo dnf install pythonPlease bare in mind that this will only install Python 2.7, which was what I needed since vscode does not support Python 3 as of the time of this writing.
         Afterwards, I also installed python’s package management system using this command:
         python get-pip.py

      4. Native-keymap
         This is a node module to support different keyboard layouts, especially since Mac keyboards have the command key, while the rest of the world has a windows key on their keyboard. To install this module, I used:
         sudo dnf install libx11-devel.x86_64 libxkbfile-devel.x86_64
         To find out more about Native-keymap, go here.

After installing the 4 key software above, I was ready to compile the vscode for the first time.  This is done with the command:
yarn run watch
My first attempt at running this command resulted in an error as per the screenshot below:

This problem was caused by the vscode file watcher running out of handles.  To fix it, I had to increased the number of handles available by editing /etc/sysctl.conf and adding the following line to the end of the file:
fs.inotify.max_user_watches=524288
This solution was found here.

After applying the fix, I managed to compile the code without errors (though the compilation took a minute or two).  Next, I attempted to test the code with the following command:
./scripts/test.sh
Fortunately, the tests ran perfectly fine for me as per the screenshot below.

Next, I ran the command to build vscode:
./scripts/code.sh

and voila!

The vscode OSS Dev was successfully built.  This concludes my First Encounter in building open source software.

Owen