Burp Extension Python Tutorial – Generate a Forced Browsing Wordlist

This post provides step by step instructions for writing a Burp Extension in Python. The extension will generate a wordlist for a specified site in the Sitemap, that can be used as a custom list for forced browsing (dirbusting) at another time. Here is what this post will cover:

• Importing required modules and accessing Burp’s interface
• Interacting with the Burp context menu (right-click and perform a function)
• Writing the functions that interact with requests and responses in the Sitemap

This is what we will be building:

Interacting with the context menu:

Output a wordlist to the Extender tab output area:

Setup

• Create a folder where you’ll store extensions – I named mine extensions
• Download the Jython standalone JAR file – Place into the extensions folder
• Download exceptions_fix.py to the extensions folder – this will make debugging easier
• Configure Burp to use Jython – Extender > Options > Python Environment > Select file
• Create a new file (GenerateForcedBrowseWordlist.py) in your favorite text editor (save it in your extensions folder)

Importing required modules, accessing the Extender API, and implementing the debugger

Hopefully everything in this tutorial will be easy to follow, but if not, the completed extension can be found here. Let’s write some code:

from burp import IBurpExtender, IContextMenuFactory
from java.util import ArrayList
from javax.swing import JMenuItem
import threading
import sys
try:
    from exceptions_fix import FixBurpExceptions
except ImportError:
    pass

The IBurpExtender module is required for all extensions, while IContextMenuFactory allows us to have the right-click functionality. The JMenuItem is used for the context menu GUI, and the ArrayList is to store our list of options that we want to appear in the context menu. The sys module is imported to allow Python errors to be shown in stdout with the help of the FixBurpExceptions script. I placed that in a Try/Except block so if we don’t have the script the code will still work fine.

This next code snippet will implement the FixBurpExceptions prettier debugger, set references to our callbacks and extension helpers, register our extension with Burp, and keep create a context menu. If you’re following along type or paste this code after the imports:

class BurpExtender(IBurpExtender, IContextMenuFactory):
    def registerExtenderCallbacks(self, callbacks):
        
        sys.stdout = callbacks.getStdout()
        self.callbacks = callbacks
        self.helpers = callbacks.getHelpers()
        self.callbacks.setExtensionName("Forced Browsing Wordlist Generator")
        callbacks.registerContextMenuFactory(self)
        
        return

try:
    FixBurpExceptions()
except:
    pass

The above class implements IBurpExtender, which is required for all extensions and must be named BurpExtender. Within the required method, registerExtendedCallbacks, the line self.callbacks keeps a reference to Burp so we can interact with it, and in our case will be used to set the extension name, and eventually obtain data from the Sitemap. The line callbacks.registerContextMenuFactory(self) tells Burp that we want to use the context menu, which is the right-click functionality. FixBurpExceptions is called at the end of the script just in case we have an error (Thanks for the code, SecurityMB!). The try/except block calling FixBurpExceptions will always go at the very end of the script.

Save the script to your extensions folder and then load the file into Burp: Extender > Extensions > Add > Extension Details > Extension Type: Python > Select file… > GenerateForcedBrowseWordlist.py

The extension should load without any errors or output. If you click on the Target > Sitemap and right-click something, if you go back to the Extender tab you should now have an error:

The error is recorded as NotImplementedError, because we invoked the iContextMenuFactory but did not implement any menu items. We can figure out why this happened by reviewing the Extender API documentation (either in Burp or online):

As shown in the above image, the method createMenuItems() “…will be called by Burp when the user invokes a context menu anywhere within Burp. The factory can then provide any custom context menu items that should be displayed in the context menu, based on the details of the menu invocation.” The documentation is telling us we need to implement the createMenuItems method and have it return menu items.

Creating an interface to right-click and perform a function

We will create the menu items, and then define the functions that are called when the menu items are clicked:

        ...
        callbacks.registerContextMenuFactory(self)
        
        return

    def createMenuItems(self, invocation):
        self.context = invocation
        menuList = ArrayList()
        menuItem = JMenuItem("Generate forced browsing wordlist from selected items",
                              actionPerformed=self.createWordlistFromSelected)
        menuList.add(menuItem)
        menuItem = JMenuItem("Generate forced browsing wordlist from all hosts in scope",
                              actionPerformed=self.createWordlistFromScope)
        menuList.add(menuItem)
        return menuList

    def createWordlistFromSelected(self, event):
        print "in createWordlistFromSelected"

    def createWordlistFromScope(self, event):
        print "in createWordlistFromScope"

try:
    FixBurpExceptions()
...

Save the code and reload the extension. Try right-clicking in the Sitemap, and you should now see the option to “Generate forced browsing wordlist from selected items” or “Generate forced browsing wordlist from all hosts in scope”. Click on one of them, and it will execute the function, and you should see output in the Extender output pane:

Excellent. So far we’ve added out menu items to the context menu, and we are able to run our functions when the menu items are clicked. The next part of the program builds out these function further, and shows how to interact with the recorded HTTP requests and responses contained in the Sitemap.

We defined two menu options, “Generate forced browsing wordlist from selected items” or “Generate forced browsing wordlist from all hosts in scope”, so we need to actually make these functions do something other than print. They are actually both going to basically do the same thing, which is start another thread and call another function that will do most of the work. The only difference between the functions will be that createWordlistFromScope() will set a class variable that tells only looks at the sites in scope. On to the code. We edit the functions that we created so they will do more than just print:


    def createMenuItems(self, invocation):
        ...
        return menuList

    def createWordlistFromSelected(self, event):
        self.fromScope = False
        t = threading.Thread(target=self.createWordlist)
        t.daemon = True
        t.start()

    def createWordlistFromScope(self, event):
        self.fromScope = True
        t = threading.Thread(target=self.createWordlist)
        t.daemon = True
        t.start()

    def createWordlist(self):
        print "In createWordlist"

try:
    FixBurpExceptions()
...

The self.fromScope variable is set so the createWordlist function will know whether to look at all of the items in scope or to only look and the site(s) that were selected in the context menu. Then, a thread is defined (t), configured to run the createWordlist function (target=self.createWordlist), and start the thread. Without multi-threading, if we try to run the extension and have a large Sitemap or multiple targets selected, then the GUI will freeze while the program is running.

If you save and reload this extension, then right-click and send the data to our extension, you should receive the following output in the Extender output tab:

Now, we can finish the createWordList() function, which is where we interact with the Sitemap:

Writing the function that interacts with requests and responses in the Sitemap

We can review the API documentation (within Burp) to see how to get the data from the Sitemap:

So if called without any parameter, the entire Sitemap is returned. If a URL prefix is specified, it will only return the Sitemap data that startswith the URL prefix. Recall that our program gives the user two options: “Generate forced browsing wordlist from selected items” or “Generate forced browsing wordlist from all hosts in scope”. To generate the wordlist for all hosts in scope we can return the entire Sitemap, and then use another Burp callbacks method isInScope() to determine whether or not we should use it. Note: When testing this extension I noticed that if you click on an out-of-scope entry in the Sitemap and select “Generate forced browsing wordlist from all hosts in scope”, it will still include that selection, as if it was in scope.

To generate the wordlist from selected items we first need to record what is selected, then we can either pull the entire Sitemap and compare the URLs we want, or we can give the getSiteMap() the URL prefix for each site individually. I chose the former for this program.

First, we determine what the user’s selection was:


    def createWordlistFromScope(self, event):
        ...
        t.start()
    
    def createWordlist(self):
        httpTraffic = self.context.getSelectedMessages()        
        hostUrls = []
        for traffic in httpTraffic:
            try:
                hostUrls.append(str(traffic.getUrl()))
            except UnicodeEncodeError:
                continue

Recall that self.context was what the user had selected in the Sitemap when they right-clicked, and the getSelectedMessages() method returns an array of objects containing data about the the items the user had selected. When I was developing the extension I inspected the object to get an idea of what it contained:

print type(httpTraffic)
<type 'array.array'>

print dir(traffic)
['...', 'b', 'class', 'comment', 'equals', 'getClass', 'getComment', 'getHighlight', 'getHost', 'getHttpService', 'getPort', 'getProtocol', 'getRequest', 'getResponse', 'getStatusCode', 'getUrl', 'hashCode', 'highlight', 'host', 'httpService', 'notify', 'notifyAll', 'port', 'protocol', 'request', 'response', 'setComment', 'setHighlight', 'setHost', 'setHttpService', 'setPort', 'setProtocol', 'setRequest', 'setResponse', 'statusCode', 'toString', 'url', 'wait']

We are interested in getting the URL from each object, so we iterate through the array and call the getUrl() method. This method returns a type of ‘java.net.URL’, which we convert to a string using str() and add it to our hostUrls list that we will use later to filter the Sitemap data. The try/except block is to deal with any encoding errors, which I handle by ignoring not adding it to the hostUrls list.

Now, we get the data from the Sitemap:


    def createWordlist(self):
        ...
                continue

        urllist = []
        siteMapData = self.callbacks.getSiteMap(None)
        for entry in siteMapData:
            requestInfo = self.helpers.analyzeRequest(entry)
            url = requestInfo.getUrl()
            try:
                decodedUrl = self.helpers.urlDecode(str(url))
            except Exception as e:
                continue

            if self.fromScope and self.callbacks.isInScope(url):
                urllist.append(decodedUrl)
            else:
                for url in hostUrls:
                    if decodedUrl.startswith(str(url)):
                        urllist.append(decodedUrl)

We initialize a new list (urllist) to hold the URLs from each site, then call getSiteMap(None), which will return all of the Sitemap entries. For each entry, we use the analyzeRequest() method to get the URL, and then URL decode each entry.

It is at this point that we get to our filtering. If self.fromScope is true, the isInScope() method is called on the URL. If that returns true, then the URL-decoded URL is appended to our urllist. If self.fromScope is False (meaning the user chose “Generate forced browsing wordlist from selected items”), the URL from the Sitemap is checked against the URLs that the user had selected in the context menu. If the decoded URL starts with the user-selected URL, then it is appended to the urllist.

Now, the urllist variable contains a list of URLs, complete with the querystring. Since we don’t need the querystring, and only want the last part of the path, we need to split up the URL and take only the part we want:


    def createWordlist(self):
        ...
                        urllist.append(decodedUrl)

        filenamelist = []
        for entry in urllist:
            filenamelist.append(entry.split('/')[-1].split('?')[0])

        for word in sorted(set(filenamelist)):
            if word:
                try:
                    print word
                except UnicodeEncodeError:
                    continue 

try:
    FixBurpExceptions()
...

The filenamelist is where we will store our forced browsing wordlist. We split each URL entry in urllist, first by the ‘/’. The split function() turns the string URL into a list, and the [-1] will grab the last element of that list. That last element will be the filename and any querystring, so it is split again at the ‘?’ and the first element is selected. For example:

>>> url = 'http://example.com/app/folder/file.php?param=value'
>>> url.split('/')
['http:', '', 'example.com', 'app', 'folder', 'file.php?param=value']
>>> url.split('/')[-1]
'file.php?param=value'
>>> url.split('/')[-1].split('?')
['file.php', 'param=value']
>>> url.split('/')[-1].split('?')[0]
'file.php'

That filename is appended into the filenamelist. Finally, we iterate through the filenamelist (after sorting and unique’ing the list) and print everything into the Extender output pane.

And that’s it! Save, reload, and you should now have a functional extension that makes use of the context menu and Sitemap. Feel free to modify any of this code to better suit your needs. For example, you may not want the JavaScript or image filenames, or maybe you want to automatically start mangling the wordlist. Or perhaps you want to do something completely different with the context menu, which is great! Hopefully this post makes developing your own extensions a little easier (or a little less intimidating).

 

Leave a Reply

Your email address will not be published. Required fields are marked *