Action logger

The Action Logger can be used to record a wide range of UI actions from multiple SW applications during the enactment of a routine.

Operating System

OS

modules.events.systemEvents.detectSelectionWindowsExplorer()

Log currently selected files in windows explorer

Every 0.8 seconds it queries File explorer to check for selection.

Returns:

selectedFile/selectedFolder event

modules.events.systemEvents.logHotkeys()

Log hotkeys using ‘keyboard’ python package.

All supported hotkeys are store in ‘keys_to_detect’ dictionary.

Once a supported hotkey is detected, a new thread is started to handle the event.

Returns:

hotkeys event

modules.events.systemEvents.logPasteHotkey()

Paste hotkey is handled separately for performance reasons. Once the hotkey is detected, a thread is started to handle the event.

Returns:

paste hotkey

modules.events.systemEvents.logProcessesMac()

detects programs opened and closed on mac.

Works by querying list of open programs every 2 seconds and determining if something has been added or removed to that list.

Returns:

modules.events.systemEvents.logProcessesWin()

Detects programs opened and closed on windows.

Works by querying list of open programs every 0.5 seconds and determining if something has been added or removed to that list.

Returns:

programOpen/programClose event

modules.events.systemEvents.logUSBDrives()

logs insertion and removal of usb drives

Returns:

insertUSB event

modules.events.systemEvents.printerLogger()

Watchfor activity at the installed printers (works also with network printers).

Returns:

printSubmitted event

modules.events.systemEvents.watchFolder()

monitor file/folder changes on windows using system APIs

https://pythonhosted.org/watchdog/api.html#event-handler-classes

Returns:

file/folder changes event

modules.events.systemEvents.watchFolderMac()

monitor file/folder changes on mac using system APIs

Returns:

file/folder changes event

modules.events.systemEvents.watchRecentsFilesWin()

Log files and folders recently opened on Windows

Returns:

openFile/openFolder event

Clipboard

modules.events.clipboardEvents.logClipboard()

Constantly monitors clipboard for changes. Detects ‘copy’ event. ‘paste’ event is detected by systemEvents.handleHotkey

Returns:

JSON containing clipboard event

Mouse

modules.events.mouseEvents.logMouse()

Log mouse coordinates on click in excel

Deprecated since version 1.1.0: Not in use anymore.

Microsoft Office

Microsoft Office events are logged using native COM APIs on windows while Excel on macOS is logged using a custom JavaScript AddIn.

Note

For each office application there are two components in the code:

• a function, triggered by the action logger and responsible for handling events

• a class, which is the interface used to communicate with the application’s APIs

Excel

modules.events.officeEvents.excelEvents(status_queue, filepath=None)

Handle excel events on windows using pythoncom.

Also manages exception in case excel instance fails to launch.

Parameters:

• status_queue – queue to pring messages on GUI

• filepath – optional path of excel file to open

modules.events.officeEvents.excelEventsMacServer(status_queue, excelFilepath=None)

Handle excel events on macOS using xlwings.

The logging process is different on macOS. An excel Add-in in javascript is installed on excel and it sends data to a node.js server started by this function.

Parameters:

• status_queue – queue to print messages on GUI

• excelFilepath – optional path of excel file to open

Returns:

openWorkbook event

class modules.events.officeEvents.ExcelEvents

Excel application object events

The parameters in each method of these class are passed automatically by the APIs when the event occurs.

https://docs.microsoft.com/en-us/office/vba/api/excel.application(object)

OnAfterCalculate()

This event occurs after a calculation has been performed

Returns:

afterCalculate event

OnNewWorkbook(Wb)

New workbook is created

Parameters:

Wb – workbook

Returns:

newWorkbook event

OnSheetActivate(Sh)

Occurs when a worksheet is selected

Parameters:

Sh – worksheet

Returns:

selectWorksheet event

OnSheetBeforeDelete(Sh)

Occurs when a worksheet is deleted

Parameters:

Sh – worksheet

Returns:

deleteWorksheet event

OnSheetBeforeDoubleClick(Sh, Target, Cancel)

Triggered when double clicking on a cell

Parameters:

• Sh – worksheet

• Target – cell clicked

• Cancel – true if event is canceled

Returns:

doubleClickCellWithValue/doubleClickEmptyCell event

OnSheetBeforeRightClick(Sh, Target, Cancel)

Triggered when right clicking on a cell

Parameters:

• Sh – worksheet

• Target – cell clicked

• Cancel – true if event is canceled

Returns:

rightClickCellWithValue/rightClickEmptyCell event

OnSheetCalculate(Sh)

Occurs when a calculation is performed

Parameters:

Sh – worksheet

Returns:

sheetCalculate event

OnSheetChange(Sh, Target)

Occurs when a cell is edited

Parameters:

• Sh – worksheet

• Target – edited cell

Returns:

editCellSheet event

OnSheetDeactivate(Sh)

Occurs when a worksheet is deselected

Parameters:

Sh – worksheet

Returns:

deselectWorksheet event

OnSheetFollowHyperlink(Sh, Target)

Occurs when a link is clicked

Parameters:

• Sh – worksheet

• Target – cell clicked

Returns:

followHiperlinkSheet event

OnSheetPivotTableAfterValueChange(Sh, TargetPivotTable, TargetRange)

Occurs when values in a pivot table change.

Parameters:

• Sh – worksheet

• TargetPivotTable – target table

• TargetRange – range modified

Returns:

pivotTableValueChangeSheet event

OnSheetSelectionChange(Sh, Target)

Occurs when a cell or range of cells is selected

Parameters:

• Sh – worksheet

• Target – selected range

Returns:

getCell/getRange event

OnSheetTableUpdate(Sh, Target)

Occurs when a table is updated

Parameters:

• Sh – worksheet

• Target – table

Returns:

worksheetTableUpdated event

OnWindowActivate(Wb, Wn)

Triggers when excel windows is opened

Parameters:

• Wb – workbook

• Wn – window

Returns:

openWindow event

OnWindowDeactivate(Wb, Wn)

Triggers when excel windows is closed

Parameters:

• Wb – workbook

• Wn – window

Returns:

closeWindow event

OnWindowResize(Wb, Wn)

Triggers when excel windows is resized

Parameters:

• Wb – workbook

• Wn – window

Returns:

resizeWindow event

OnWorkbookActivate(Wb)

workbook is activated (occurs when excel window is clicked and goes to foreground)

Parameters:

Wb – workbook

Returns:

activateWorkbook event

OnWorkbookAddinInstall(Wb)

An AddIn is installed in excel

Parameters:

Wb – workbook

Returns:

addinInstalledWorkbook event

OnWorkbookAddinUninstall(Wb)

An AddIn is uninstalled in excel

Parameters:

Wb – workbook

Returns:

addinUninstalledWorkbook event

OnWorkbookAfterSave(Wb, Success)

This event is triggered after a workbook has been saved.

Parameters:

• Wb – workbook

• Success – boolean value to indicate if saving was succesful

Returns:

saveWorkbook event

OnWorkbookAfterXmlExport(Wb, Map, Url, Result)

XML is exported from excel

Parameters:

• Wb – workbook

• Map –

• Url –

• Result – import result

Returns:

XMLExportWorkbook event

OnWorkbookAfterXmlImport(Wb, Map, Url, Result)

XML is imported into excel

Parameters:

• Wb – workbook

• Map –

• Url –

• Result – import result

Returns:

XMLImportWorkbook event

OnWorkbookBeforeClose(Wb, Cancel)

workbook is closed

Parameters:

• Wb – workbook

• Cancel – true if close is canceled

Returns:

closeWorkbook event

OnWorkbookBeforePrint(Wb, Cancel)

Print action is triggered in a workbook

Parameters:

• Wb – workbook

• Cancel – true if print is canceled

Returns:

printWorkbook event

OnWorkbookBeforeSave(Wb, SaveAsUI, Cancel)

This is triggered right before a workbook is saved

Parameters:

• Wb – workbook

• SaveAsUI – ‘save as’ dialog box displayed to user

• Cancel – saving is canceled

Returns:

beforeSaveWorkbook event

OnWorkbookDeactivate(Wb)

workbook is deactivated (occurs when excel window is not visible anymore)

Parameters:

Wb – workbook

Returns:

deactivateWorkbook event

OnWorkbookNewChart(Wb, Ch)

A new chart is added to the open workbook

Parameters:

• Wb – workbook

• Ch – chart

Returns:

newChartWorkbook event

OnWorkbookNewSheet(Wb, Sh)

New sheet is created

Parameters:

• Wb – workbook

• Sh – worksheet

Returns:

addWorksheet event

OnWorkbookOpen(Wb)

Workbook is opened

Parameters:

Wb – workbook

Returns:

openWorkbook event

__init__()

filterNoneRangeValues(values)

When a range of cells is selected, Target.Value is a Tuple of tuples containing the value of every selected cell Target.Value = ((None, None, None, None), (None, ‘prova’, None, None)). I’m interested only in cells with meaningful value, so I create a single list with all the tuples in Target.Value (by chaining the tuples using chain.from_iterable(list)) obtaining [None, None, None, None, None, ‘prova’, None, None] Now I remove the elements that are None by applying a filter operator to the previous list

Parameters:

values – tuple of values

Returns:

tuple of values without None values

getWorksheets(Sh, Wb)

return list of active worksheet in workbook

Parameters:

• Sh – worksheet

• Wb – workbook

Returns:

list of active worksheet in workbook

Word

modules.events.officeEvents.wordEvents(filename=None)

Handle word events on windows using pythoncom.

Parameters:

filename – optional name of file to open

class modules.events.officeEvents.WordEvents

Application object events https://docs.microsoft.com/en-us/office/vba/api/word.application

OnDocumentBeforePrint(Doc, Cancel)

Occurs when a document is printed

Parameters:

• Doc – document

• Cancel – true if event is canceled

Returns:

saveDocument event

OnDocumentBeforeSave(Doc, SaveAsUI, Cancel)

Occurs when a document is saved

Parameters:

• Doc – document

• SaveAsUI – true if ‘save as’ dialog is displayed

• Cancel – true if event is canceled

Returns:

saveDocument event

OnDocumentChange()

Occurs when a document is changed

Returns:

changeDocument event

OnDocumentOpen(Doc)

Occurs when a new document is opened

Parameters:

Doc – document

Returns:

openDocument event

OnNewDocument(Doc)

Occurs when a new document is created

Parameters:

Doc – document

Returns:

newDocument event

OnWindowActivate(Doc, Wn)

Occurs when window is activated

Parameters:

• Doc – document

• Wn – window

Returns:

activateWindow event

OnWindowBeforeDoubleClick(Sel, Cancel)

Occurs when double clicking on document

https://docs.microsoft.com/en-us/office/vba/api/word.selection#properties

Parameters:

• Sel – selection

• Cancel – true if action is canceled

Returns:

doubleClickWindow event

OnWindowBeforeRightClick(Sel, Cancel)

Occurs when right clicking on document

Parameters:

• Sel – selection

• Cancel – true if action is canceled

Returns:

rightClickWindow event

OnWindowDeactivate(Doc, Wn)

Occurs when window is deactivated

Parameters:

• Doc – document

• Wn – window

Returns:

activateWindow event

__init__()

PowerPoint

modules.events.officeEvents.powerpointEvents(filename=None)

Handle powerpoint events on windows using pythoncom.

Parameters:

filename – optional name of file to open

class modules.events.officeEvents.PowerpointEvents

Application object events https://docs.microsoft.com/en-us/office/vba/api/powerpoint.application

OnAfterPresentationOpen(Pres)

Occurs when a presentation is opened

Parameters:

Pres – presentation

Returns:

savePresentation event

OnAfterShapeSizeChange(shp)

Occurs when a shape in a presentation is resized

Parameters:

shp – shape

Returns:

shapeSizeChangePresentation event

OnNewPresentation(Pres)

New presentation is created

Parameters:

Pres – presentation

Returns:

newPresentation event

OnPresentationBeforeClose(Pres, Cancel)

Occurs when a presentation is closed

Parameters:

• Pres – presentation

• Cancel – true if event is canceled

Returns:

closePresentation event

OnPresentationBeforeSave(Pres, Cancel)

Occurs when a presentation is saved

Parameters:

• Pres – presentation

• Cancel – true if event is canceled

Returns:

savePresentation event

OnPresentationNewSlide(Sld)

A slide is added to an open presentation

Parameters:

Sld – slide

Returns:

newPresentationSlide event

OnPresentationPrint(Pres)

Occurs when a presentation is printed

Parameters:

Pres – presentation

Returns:

printPresentation event

OnSlideSelectionChanged(SldRange)

occurs when text selection on a slide changes

Parameters:

SldRange – range of slides

Returns:

SlideSelectionChanged event

OnSlideShowBegin(Wn)

Occurs when slideshow begins

Parameters:

Wn – slideshowview https://docs.microsoft.com/en-us/office/vba/api/powerpoint.slideshowview

Returns:

slideshowBegin event

OnSlideShowEnd(Pres)

Occurs when a slideshow ends

Parameters:

Pres – presentation

Returns:

slideshowEnd event

OnSlideShowNextClick(Wn, nEffect)

Occurs when the next slide in a presentation is clicked during a slideshow

https://docs.microsoft.com/en-us/office/vba/api/powerpoint.effect#properties

Parameters:

• Wn – slideshowview https://docs.microsoft.com/en-us/office/vba/api/powerpoint.slideshowview

• nEffect – transition type

Returns:

clickNextSlideshow event

OnSlideShowOnNext(Wn)

Occurs when next slide is displayed during a slideshow

Parameters:

Wn – slideshowview https://docs.microsoft.com/en-us/office/vba/api/powerpoint.slideshowview

Returns:

nextSlideshow

OnSlideShowOnPrevious(Wn)

Occurs when previous slide is clicked during a slideshow

https://docs.microsoft.com/en-us/office/vba/api/powerpoint.slideshowview.state

Parameters:

Wn – slideshowview https://docs.microsoft.com/en-us/office/vba/api/powerpoint.slideshowview

Returns:

previousSlideshow event

OnWindowActivate(Pres, Wn)

Occurs when powerpoint window is activated

Parameters:

• Pres – powerpoint presentation

• Wn – window

Returns:

activateWindow event

OnWindowBeforeDoubleClick(Sel, Cancel)

Occurs on right click on presentation

Parameters:

• Sel – selection

• Cancel – true if event is canceled

Returns:

doubleClickPresentation event

OnWindowBeforeRightClick(Sel, Cancel)

Occurs on right click on presentation

Parameters:

• Sel – selection

• Cancel – true if event is canceled

Returns:

rightClickPresentation event

OnWindowDeactivate(Pres, Wn)

Occurs when powerpoint window is deactivated

Parameters:

• Pres – powerpoint presentation

• Wn – window

Returns:

deactivateWindow event

__init__()

Outlook

modules.events.officeEvents.outlookEvents()

Handle outlook events using pythoncom

class modules.events.officeEvents.OutlookEvents

https://stackoverflow.com/questions/49695160/how-to-continuously-monitor-a-new-mail-in-outlook-and-unread-mails-of-a-specific

OnItemSend(Item, Cancel)

Occurs when a new message is sent

Parameters:

• Item – message

• Cancel – true if action is canceled

Returns:

sendMail event

OnMAPILogonComplete()

Occurs when outlook checks for new mails

Returns:

logonComplete event

OnNewMailEx(receivedItemsIDs)

Occurs when a new mail is received

Parameters:

receivedItemsIDs – id of received mail

Returns:

receiveMail event

OnQuit()

Occurs when outlook quits

Returns:

quitOutlook event

OnReminder(Item)

Occurs when there is a new reminder

Parameters:

Item – reminder item

Returns:

newReminder event

OnStartup()

Occurs when outlook starts

Returns:

startupOutlook event

__init__()

Browser

Note

Browser events are handled in the browser extension written in JavaScript.

In particular:

• Browser application events (e.g. newTab, zoomTab, resizeWindow, …) are handled in extensions/browserLogger/background_script.js

• Web page events (e.g. click, type, submit, …) are handled in extensions/browserLogger/content_script.js

Logging Server

The logging server was built using Flask. The server is launched when the user presses the Start Logger button in the GUI.

The selected browsers that needs to be enabled and the log file path are passed as parameters. At launch, the program checks if port 4444 is free on the system: in this case it starts listening for incoming requests.

The default route (/) is used to receive JSON payloads by every module containing data about an event and write them on the CSV file. This payload contains different fields depending on the type of recorded event. When the program is started, the CSV log file is created and the header is written.

modules.consumerServer.add_headers(response)

Enable CORS, for browser extension

https://stackoverflow.com/a/35306327

modules.consumerServer.getServerStatus()

Get server status for browser extension.

Returns status of each browser checkbox in GUI.

Returns:

true if browser checkbox in GUI is active

modules.consumerServer.runServer(status_queue)

start server thread, executed by mainLogger

Parameters:

status_queue – queue to print messages in GUI

modules.consumerServer.writeLog()

Route where json event is received and processed.

JSON event includes metadata about the event, such as the timestamp, category, application, concept:name and other information depending on the event type.

All this data is appended to the csv event log.