AXElements 0.6.0beta1
Sign up to get free protection for your applications and to get access to all the features.
- data/.yardopts +20 -0
- data/LICENSE.txt +25 -0
- data/README.markdown +150 -0
- data/Rakefile +109 -0
- data/docs/AccessibilityTips.markdown +119 -0
- data/docs/Acting.markdown +340 -0
- data/docs/Debugging.markdown +326 -0
- data/docs/Inspecting.markdown +255 -0
- data/docs/KeyboardEvents.markdown +57 -0
- data/docs/NewBehaviour.markdown +151 -0
- data/docs/Notifications.markdown +271 -0
- data/docs/Searching.markdown +250 -0
- data/docs/TestingExtensions.markdown +52 -0
- data/docs/images/AX.png +0 -0
- data/docs/images/all_the_buttons.jpg +0 -0
- data/docs/images/ui_hierarchy.dot +34 -0
- data/docs/images/ui_hierarchy.png +0 -0
- data/ext/key_coder/extconf.rb +6 -0
- data/ext/key_coder/key_coder.m +77 -0
- data/lib/ax_elements/accessibility/enumerators.rb +104 -0
- data/lib/ax_elements/accessibility/language.rb +347 -0
- data/lib/ax_elements/accessibility/qualifier.rb +73 -0
- data/lib/ax_elements/accessibility.rb +164 -0
- data/lib/ax_elements/core.rb +541 -0
- data/lib/ax_elements/element.rb +593 -0
- data/lib/ax_elements/elements/application.rb +88 -0
- data/lib/ax_elements/elements/button.rb +18 -0
- data/lib/ax_elements/elements/radio_button.rb +18 -0
- data/lib/ax_elements/elements/row.rb +30 -0
- data/lib/ax_elements/elements/static_text.rb +17 -0
- data/lib/ax_elements/elements/systemwide.rb +46 -0
- data/lib/ax_elements/inspector.rb +116 -0
- data/lib/ax_elements/macruby_extensions.rb +255 -0
- data/lib/ax_elements/notification.rb +37 -0
- data/lib/ax_elements/version.rb +9 -0
- data/lib/ax_elements.rb +30 -0
- data/lib/minitest/ax_elements.rb +19 -0
- data/lib/mouse.rb +185 -0
- data/lib/rspec/expectations/ax_elements.rb +15 -0
- data/test/elements/test_application.rb +72 -0
- data/test/elements/test_row.rb +27 -0
- data/test/elements/test_systemwide.rb +38 -0
- data/test/helper.rb +119 -0
- data/test/test_accessibility.rb +127 -0
- data/test/test_blankness.rb +26 -0
- data/test/test_core.rb +448 -0
- data/test/test_element.rb +939 -0
- data/test/test_enumerators.rb +81 -0
- data/test/test_inspector.rb +121 -0
- data/test/test_language.rb +157 -0
- data/test/test_macruby_extensions.rb +303 -0
- data/test/test_mouse.rb +5 -0
- data/test/test_search_semantics.rb +143 -0
- metadata +219 -0
@@ -0,0 +1,340 @@
|
|
1
|
+
# Actions (a.k.a. The Fun Part)
|
2
|
+
|
3
|
+
Actions are how you drive the user interface. They can be anything
|
4
|
+
from showing a menu to dragging a window, but are often simple things
|
5
|
+
like pressing a button. Most actions are subtly different cases
|
6
|
+
of the same thing, but it is important to understand the differences
|
7
|
+
and when to use different actions.
|
8
|
+
|
9
|
+
There are three types of actions, those provided by the accessibility
|
10
|
+
API as direct calls, those provided by accessibility APIs as keyboard
|
11
|
+
simulation, and those that are provided by the CGEvents API. In most
|
12
|
+
cases you can use at least two types of actions to achieve the desired
|
13
|
+
effect and each type of action has its own pros and cons to consider.
|
14
|
+
|
15
|
+
## Interface
|
16
|
+
|
17
|
+
Though there are three different APIs under the hood, they are exposed
|
18
|
+
through a common interface via the DSL methods in
|
19
|
+
{Accessibility::Language}.
|
20
|
+
|
21
|
+
app = Accessibility.application_with_name 'Terminal'
|
22
|
+
press app.main_window.minimize_button
|
23
|
+
type "\\CMD+M", app
|
24
|
+
click app.main_window.minimize_button
|
25
|
+
|
26
|
+
The above example demonstrates performing actions using the DSL
|
27
|
+
methods. The DSL methods are always some kind of action, such as
|
28
|
+
{Accessibility::Language#drag_mouse_to drag\_mouse\_to}, or
|
29
|
+
{Accessibility::Language#set_focus set\_focus} to name two.
|
30
|
+
|
31
|
+
The point of the DSL is to put the actions at the front of statements
|
32
|
+
in order to separate verbs from nouns. In this way, scripts should
|
33
|
+
resemble instructions that you might give for reproduction steps in a
|
34
|
+
bug report---they should read like they were instructions for
|
35
|
+
a person. This style should make it easier to think about writing
|
36
|
+
scripts and it should also give better hints at when to refactor
|
37
|
+
code into helper methods like `log_in_as:with_password:`.
|
38
|
+
|
39
|
+
If you tried out the code snippet above you might notice the
|
40
|
+
difference between the APIs (you also might notice that the second
|
41
|
+
command does not work, it is a work in progress :(). In the first
|
42
|
+
case, `press` was using the actions provided by the accessibility APIs
|
43
|
+
to call the press action on the minimize button for the window, in the
|
44
|
+
second case we used `type` to type in the hot key for minimizing a
|
45
|
+
window, and in the third case we used `click` from the CGEvents APIs
|
46
|
+
to actually move the mouse cursor to the button and generate a click
|
47
|
+
event.
|
48
|
+
|
49
|
+
## Accessibility Actions
|
50
|
+
|
51
|
+
Actions like `press` use the accessibility APIs and show up in the
|
52
|
+
list of actions returned by {AX::Element#actions}. These actions will
|
53
|
+
also show up in the Accessibility Inspector in the Actions section.
|
54
|
+
|
55
|
+
You may have noticed that {Accessibility::Language#press press}
|
56
|
+
shows up as `AXPress` for buttons when you view them with the
|
57
|
+
Accessibility Inspector or by calling {AX::Element#actions}. This is
|
58
|
+
because we do name translation with accessibility actions just like
|
59
|
+
when accessing attributes as shown in the
|
60
|
+
{file:docs/Inspecting.markdown Inspection tutorial}.
|
61
|
+
|
62
|
+
Accessibility actions are the simplest and fastest way to trigger
|
63
|
+
actions. They use the accessibility APIs to talk directly to the
|
64
|
+
application to call the same methods that would be called when
|
65
|
+
you interact with the application with the mouse or keyboard, but
|
66
|
+
without having to use the mouse or keyboard.
|
67
|
+
|
68
|
+
## Setting Attributes
|
69
|
+
|
70
|
+
Though it is not really an action, some attributes can have their
|
71
|
+
values changed through the accessibility APIs. The
|
72
|
+
{Accessibility::Language} module has two methods, `set` and
|
73
|
+
`set_focus`, which allow you to change the value for certain
|
74
|
+
attributes.
|
75
|
+
|
76
|
+
In the Accessibility Inspector, a writable attribute will have `(W)`
|
77
|
+
next to the attribute name, and the programmatic way to tell if an
|
78
|
+
attribute is writable is to call {AX::Element#attribute_writable?} and
|
79
|
+
pass the name of the attribute as a parameter:
|
80
|
+
|
81
|
+
app.main_window.attribute_writable? :size
|
82
|
+
app.main_window.attribute_writable? :title
|
83
|
+
|
84
|
+
You can only set an attribute if it is writable. When you have an
|
85
|
+
attribute that is writable, and you would like to change it, then you
|
86
|
+
simply need to call `set`:
|
87
|
+
|
88
|
+
set app.main_window, size: [500, 500].to_size
|
89
|
+
|
90
|
+
The first parameter is the element, and the second parameter is a
|
91
|
+
key-value pair with the attribute as the key and the value as the
|
92
|
+
new value.
|
93
|
+
|
94
|
+
`set_focus` is just syntactic sugar for `set` where the key-value is
|
95
|
+
set for you and the only parameter you provide is the element you want
|
96
|
+
to set focus to:
|
97
|
+
|
98
|
+
set_focus app.main_window.text_field
|
99
|
+
|
100
|
+
Though, `set` itself has a special case. The second most frequently
|
101
|
+
changed attribute for a element is the `value`, it is used almost as
|
102
|
+
much as `focused`. For this case, you can pass anything that is not a
|
103
|
+
key-value pair to `set` and it will be assumed that you want to change
|
104
|
+
the value of the `value` attribute:
|
105
|
+
|
106
|
+
set app.main_window.text_field, 'Mark Rada'
|
107
|
+
|
108
|
+
Another important detail that you might be curious about is that we
|
109
|
+
called `to_size` on the first snippet showing how to set an
|
110
|
+
attribute. The developers of MacRuby have added a feature that allows
|
111
|
+
you to pass an array as an argument when you would normally be
|
112
|
+
expected to pass a structure such as a `CGPoint`. Since AXElements
|
113
|
+
APIs inherently need to handle any type of object, it is not sane to try
|
114
|
+
and do the same type of run time type analysis. In its place, I have
|
115
|
+
provided some convenience methods for {NSArray arrays} to let you
|
116
|
+
quickly transform them to the appropriate type of object. In my
|
117
|
+
opinion, you should just use the proper type of object in the first
|
118
|
+
place and avoid the overhead.
|
119
|
+
|
120
|
+
### You Want To Use Actions
|
121
|
+
|
122
|
+
The advantage to using accessibility actions is that they are usually
|
123
|
+
much easier to use than the other APIs. Accessibility actions are
|
124
|
+
usually synchronous and so your scripts will not need to worry about
|
125
|
+
waiting for animations to complete. The only complication is that
|
126
|
+
sometimes an action is not synchronous and there is no good way to
|
127
|
+
tell if an action is synchronous without looking at the underlying
|
128
|
+
code.
|
129
|
+
|
130
|
+
### Asynchronous Problems
|
131
|
+
|
132
|
+
Problems with synchronous behaviour will occur with all the types of
|
133
|
+
actions, but less often with accessibility actions, so you should
|
134
|
+
always try thing out first in the console to make sure. A good rule of
|
135
|
+
thumb is that anything that looks like it would take a noticeable
|
136
|
+
amount of time, and is not an animation, will probably be
|
137
|
+
asynchronous. An example of this would be having to load data from a
|
138
|
+
database (read: separate process, across a network, etc.) such as in
|
139
|
+
the Marketcircle server admin apps or the Billings Pro preferences.
|
140
|
+
|
141
|
+
Asynchronous behaviour can often be worked around by using
|
142
|
+
accessibility {file:docs/Notifications.markdown notifications} or by
|
143
|
+
simply calling
|
144
|
+
[`sleep`](http://rdoc.info/stdlib/core/1.9.2/Kernel#sleep-instance_method).
|
145
|
+
Notifications have some interesting caveats that are covered in their
|
146
|
+
own tutorial and so it is much easier to `sleep`.
|
147
|
+
|
148
|
+
However, using `sleep` to solve the problem of asynchronous waiting is
|
149
|
+
like using a steak knife for brain surgery. Unless you can control the
|
150
|
+
environment when the script is running, you will need to sleep for
|
151
|
+
longer periods of time than you really need to; even when you have
|
152
|
+
quite a bit of control you might still have the occasional instance
|
153
|
+
where a database fetch takes longer than expected. I would suggest you
|
154
|
+
use notifications when you can, and sleep when you cannot.
|
155
|
+
|
156
|
+
## CGEvents Actions
|
157
|
+
|
158
|
+
CGEvents based actions never directly use the accessibility APIs and
|
159
|
+
are implemented at a different level of the GUI stack in OS X (I
|
160
|
+
assume). An action that uses CGEvents will actually move the mouse
|
161
|
+
cursor around and simulate mouse input at a relatively low level of
|
162
|
+
abstraction. However, accessibility information is still used to find
|
163
|
+
the point on the screen to move to and click. These types of actions
|
164
|
+
are more realistic, more awesome looking, and also more difficult to
|
165
|
+
write.
|
166
|
+
|
167
|
+
The difficulty in writing the scripts comes from the fact that it does
|
168
|
+
not directly communicate with the accessibility APIs. This implicitly
|
169
|
+
means that all CGEvents actions are asynchronous, which is the only
|
170
|
+
non-trivial complication with using CGEvents APIs.
|
171
|
+
|
172
|
+
### CGEvents Goes Where Accessibility Cannot
|
173
|
+
|
174
|
+
The benefit of actually moving the mouse will often outweigh the
|
175
|
+
downside here. Moving the mouse cursor and generating click events
|
176
|
+
allows things like dragging and scrolling which cannot be done using
|
177
|
+
the accessibility APIs.
|
178
|
+
|
179
|
+
As mentioned earlier, CGEvents doesn't talk to applications in the
|
180
|
+
same way that accessibility APIs do, which means that you can use
|
181
|
+
CGEvents actions on elements that do not fully support
|
182
|
+
accessibility. For instance, text might appear on the UI and have a
|
183
|
+
`click`-able link but may not provide an equivalent action to clicking
|
184
|
+
on the link; using CGEvents APIs you will only need to move to the
|
185
|
+
position of the text and then generate a click event.
|
186
|
+
|
187
|
+
### Realism
|
188
|
+
|
189
|
+
Since CGEvents actually manipulates the mouse cursor, any script that
|
190
|
+
uses the APIs will be more realistic than their equivalent
|
191
|
+
accessibility actions. This can make a difference if you are testing
|
192
|
+
an app and you are expecting other side effects to occur. For instance,
|
193
|
+
what if you implemented a sort of UI element that requires the mouse
|
194
|
+
cursor to be near the element for an action to trigger, such as an
|
195
|
+
expanding drawer or a folder in the finder. Depending on what you are
|
196
|
+
using AXElements for, this may or may not be important to you.
|
197
|
+
|
198
|
+
### Underpinnings
|
199
|
+
|
200
|
+
An important thing to note is that AXElements always works with
|
201
|
+
flipped co-ordinates. The origin, `(0, 0)` is always located in the
|
202
|
+
top left corner of the main display. Displays to the left have a
|
203
|
+
negative `x` co-ordinate and displays above the main display will have
|
204
|
+
negative `y` co-ordinates.
|
205
|
+
|
206
|
+
Though the CGEvents APIs are exposed to AXElements through the
|
207
|
+
{Accessibility::Language Language} module, there is another thin
|
208
|
+
layer between AXElements and CGEvents that abstracts everything that
|
209
|
+
AXElements uses.
|
210
|
+
|
211
|
+
The between layer is the {Mouse} module, and it is responsible for
|
212
|
+
generating events and animating the mouse movements on screen. The API
|
213
|
+
that it makes available to AXElements is more fine grained than what
|
214
|
+
AXElements makes available to you. There is a lot of room for the
|
215
|
+
{Mouse} module to grow and add new features, some of which are noted
|
216
|
+
as `@todo` items in the documentation. It may also be necessary in
|
217
|
+
the future to make available from AXElements certain options that are
|
218
|
+
normally hidden in {Mouse}.
|
219
|
+
|
220
|
+
### Where The Mouse Moves
|
221
|
+
|
222
|
+
Consider the following:
|
223
|
+
|
224
|
+
move_mouse_to app.main_window.close_button
|
225
|
+
|
226
|
+
If you try out that code snippet then you will notice that the mouse
|
227
|
+
cursor moves to the button, specifically the center of the button. If
|
228
|
+
you use the method to move to another object, such as a button, you
|
229
|
+
will again notice that it moved to the center point of the element.
|
230
|
+
|
231
|
+
You can call `move_mouse_to` and give it a UI element, but you could
|
232
|
+
also be more exact and give it a CGPoint or even an array with two
|
233
|
+
numbers. In fact, you could pass just any object as long it implements
|
234
|
+
a method named `to_point` which returns a CGPoint object. If you were
|
235
|
+
to look at the code for {Accessibility::Language#move_mouse_to} you
|
236
|
+
would see that all it does is call `#to_point` on the argument and
|
237
|
+
then pass the returned value to the {Mouse} module to actually do the
|
238
|
+
work. You could also look at {NSArray#to_point} and find out that it just
|
239
|
+
returns a new CGPoint using the first two objects in the array, and
|
240
|
+
`CGPoint#to_point` just returns itself (you can't see the
|
241
|
+
documentation for the method because of limitations in
|
242
|
+
YARD). Similarly, {AX::Element} implements
|
243
|
+
{AX::Element#to_point to_point} which not only gets the position for
|
244
|
+
the element, but also gets the size and then calculates the center
|
245
|
+
point for the element. This is important because you probably never
|
246
|
+
want the mouse cursor moving to the top left of the UI element, but
|
247
|
+
maybe you don't want the cursor moving to the center either.
|
248
|
+
|
249
|
+
If you want to have the mouse move to a location over a UI element
|
250
|
+
that is not the center point, then you will need to implement
|
251
|
+
`to_point` for the appropriate class. Just remember to follow the
|
252
|
+
{file:docs/NewBehaviour.markdown rules} for choosing the proper
|
253
|
+
superclasss. For instance, on Snow Leopard you could resize a window
|
254
|
+
by clicking and dragging an `AX::GrowArea` object in the lower right
|
255
|
+
corner of a window; moving to the center of the element may not
|
256
|
+
actually allow the mouse to respond and so you would have to implement
|
257
|
+
`to_point` to move closer to the bottom right of the window
|
258
|
+
(__NOTE__: I don't know if that is actually true for `AX::GrowArea`,
|
259
|
+
it was just meant as an example).
|
260
|
+
|
261
|
+
### Dragging
|
262
|
+
|
263
|
+
Sometimes you just need to be able to click and drag an element. To do
|
264
|
+
this you can only use the CGEvents APIs. Fortunately, click and drag
|
265
|
+
is relatively painless with AXElements, the simplest example would be
|
266
|
+
something like this:
|
267
|
+
|
268
|
+
# move to the mouse to the starting point
|
269
|
+
move_mouse_to app.main_window.title_ui_element
|
270
|
+
|
271
|
+
# start the click and drag event
|
272
|
+
drag_mouse_to [0, 0]
|
273
|
+
|
274
|
+
Pretty cool, eh? The general pattern is to move the mouse to the
|
275
|
+
starting point, and then to call `drag_mouse_to` to start the click
|
276
|
+
and drag events. In the example, I gave co-ordinates using an array
|
277
|
+
with two numbers but you can pass a CGPoint or anything else that
|
278
|
+
responds to `to_point` just like with the other CGEvents actions.
|
279
|
+
|
280
|
+
## Keyboard Actions
|
281
|
+
|
282
|
+
The final type of action is the keyboard action. Keyboard actions
|
283
|
+
live in a world between the accessibility APIs and CGEvent APIs; that
|
284
|
+
is, Apple has already done the work of unifying them for me and in a
|
285
|
+
way that I would not have been able to do myself.
|
286
|
+
|
287
|
+
The keyboard action is a single method,
|
288
|
+
{Accessibility::Language#type}, that requires one parameter and
|
289
|
+
takes an optional second parameter. The first parameter is simply a
|
290
|
+
string that includes which characters you wish to have typed out and
|
291
|
+
the second parameter is the UI element for the application where you
|
292
|
+
would like to send the keyboard events. Since the second parameter is
|
293
|
+
optional, the currently focused app will receive the events if the
|
294
|
+
parameter is not used.
|
295
|
+
|
296
|
+
# to a specific app
|
297
|
+
app = Accessibility.application_with_name 'Terminal'
|
298
|
+
type "Hello, world!", app
|
299
|
+
|
300
|
+
# to the focused app
|
301
|
+
type "Hello, world!"
|
302
|
+
|
303
|
+
The string that you pass can have some special escape sequences to
|
304
|
+
indicate that you want to press command characters or a combination of
|
305
|
+
keys at once for a hot key. Details on this are contained in their own
|
306
|
+
tutorial, the
|
307
|
+
{file:docs/KeyboardEvents.markdown Keyboard Events Tutorial}.
|
308
|
+
|
309
|
+
## Mixing In
|
310
|
+
|
311
|
+
By now you will have noticed that all the actions have been defined in
|
312
|
+
the {Accessibility::Language} name space, which is just a simple
|
313
|
+
module. Though, you have been able to use the methods anywhere and in
|
314
|
+
any context.
|
315
|
+
|
316
|
+
When you load AXElements, by using `require 'ax_elements'`, not only
|
317
|
+
will all the code be loaded, but the extra step of mixing
|
318
|
+
{Accessibility::Language} into the top level name space will also be
|
319
|
+
done for you. The only way to override this behaviour is to load the
|
320
|
+
components of AXElements yourself, the details on how to do this are
|
321
|
+
left as an exercise to interested parties. :P
|
322
|
+
|
323
|
+
## A Note About Caching
|
324
|
+
|
325
|
+
You need to be careful when you cache UI element objects. Every time
|
326
|
+
that you trigger an action you are intrinsically changing the state of
|
327
|
+
an application. State changes will often cause new UI elements to be
|
328
|
+
created, recreated, or removed.
|
329
|
+
|
330
|
+
For example consider pressing the close button for a window; in this
|
331
|
+
case, an entire window and all of its children will disappear and
|
332
|
+
become invalid objects. Another case might be pressing the `+` button
|
333
|
+
for a table; in this case you have created a new row for the table and
|
334
|
+
any cache of the existing rows for the table will not include the new
|
335
|
+
element.
|
336
|
+
|
337
|
+
The real problem with caching is with the invalid objects. An invalid
|
338
|
+
object is poisonous to the MacRuby run time. If you try to access an
|
339
|
+
attribute or trigger a search from an invalid then you will cause
|
340
|
+
MacRuby to crash.
|
@@ -0,0 +1,326 @@
|
|
1
|
+
# Debugging
|
2
|
+
|
3
|
+
This document includes instructions on using AXElements' built in
|
4
|
+
tools to help you debug issues with your scripts, or in cases where
|
5
|
+
you might find a bug in AXElements itself or MacRuby.
|
6
|
+
|
7
|
+
## Trees
|
8
|
+
|
9
|
+
Sometimes you need to see the big picture, the whole UI tree at
|
10
|
+
once or at least be able to see the root of the hierarchy from where
|
11
|
+
you are. For these troubling cases AXElements provides a few tools.
|
12
|
+
|
13
|
+
### Text Tree
|
14
|
+
|
15
|
+
Printing a text tree is similar to how a UI dump works with
|
16
|
+
accessibility on iOS. AXElements does improve a bit on its iOS
|
17
|
+
equivalent. Simply using the Accessibility Inspector does not give you
|
18
|
+
a good picture of the whole tree and with complicated structures it is
|
19
|
+
easy to make mistakes navigating the UI tree.
|
20
|
+
|
21
|
+
The text tree comes formatted for you, and you can simply print it out
|
22
|
+
to the console. The output uses indentation to indicate how far down
|
23
|
+
the tree each element is, and the first element up the tree with one
|
24
|
+
indentation level less will always be the parent for an element.
|
25
|
+
|
26
|
+
Printing out a text tree in the console is very easy. First you generate
|
27
|
+
the text dump and then you print it out:
|
28
|
+
|
29
|
+
puts Accessibility.dump(app)
|
30
|
+
|
31
|
+
This method is useful when you are having difficulties with
|
32
|
+
search. Sometimes when searching you will end up finding the wrong
|
33
|
+
element because your search query was ambiguous and you didn't
|
34
|
+
know at the time. Printing out a dump can often be all the help you
|
35
|
+
need in order to identify your problem.
|
36
|
+
|
37
|
+
However, if the standard `#inspect` isn't doing it for you, you can
|
38
|
+
perform your own inspection every element in a tree yourself using the
|
39
|
+
built in enumerators. Searches use the {Accessibility::BFEnumerator},
|
40
|
+
but the text tree dump method uses {Accessibility::DFEnumerator}.
|
41
|
+
|
42
|
+
### Dot Graph
|
43
|
+
|
44
|
+
__NOTE__: This feature isn't actually done yet. Coming soon, I promise.
|
45
|
+
|
46
|
+
For super fancy text trees, AXElements can generate dot graphs for
|
47
|
+
consumption by [Graphviz](http://www.graphviz.org/). In this case, you
|
48
|
+
want to call {Accessibility.graph} and pass the root of the tree you
|
49
|
+
want to have turned into a dot graph; you will get a string back that
|
50
|
+
you will then need to give to Graphviz in order to generate the visual
|
51
|
+
graph.
|
52
|
+
|
53
|
+
### Text Tree Won't Print?
|
54
|
+
|
55
|
+
AXElements isn't perfect and it is possible that an edge case slipped
|
56
|
+
between the cracks. However, it's also possible that the problem is
|
57
|
+
actually how accessibility information is being vended out.
|
58
|
+
|
59
|
+
As an example, Radar #10040865 is a ticket that I filed with Apple
|
60
|
+
where they broke accessibility with search fields. The effect to
|
61
|
+
AXElements was that you could not search through the menu bar causing
|
62
|
+
a text dump to fail in a very difficult to trace manner.
|
63
|
+
|
64
|
+
In these cases you will need to use your deductive reasoning to figure
|
65
|
+
out where the problem is coming from. Fortunately, I have provided
|
66
|
+
some tools to help you along the way.
|
67
|
+
|
68
|
+
## All The Way Up
|
69
|
+
|
70
|
+
Sometimes you don't need a whole tree to be printed out. Sometimes
|
71
|
+
just seeing the path from an element to the top level element is
|
72
|
+
enough. {Accessibility.path} is your friend in this case, it will
|
73
|
+
provide you with an array of UI element objects, each successive
|
74
|
+
element will be the parent of the previous element. This is almost
|
75
|
+
like the view that the Accessibility Inspector provides.
|
76
|
+
|
77
|
+
## Custom Exceptions
|
78
|
+
|
79
|
+
AXElements provides some customized exceptions in the OO layer that
|
80
|
+
should help give you much better hints at what went wrong when you
|
81
|
+
have a problem.
|
82
|
+
|
83
|
+
Custom exceptions have been created to help identify the point of
|
84
|
+
failure that would have caused a more cryptic exception to have been
|
85
|
+
raised instead. These custom exceptions also capture more metadata
|
86
|
+
about the problem that occurred which should give good hints as to what
|
87
|
+
went wrong.
|
88
|
+
|
89
|
+
### Search Failures
|
90
|
+
|
91
|
+
An {AX::Element::SearchFailure `SearchFailure`} will occur when you
|
92
|
+
perform an implicit search that fails to find anything.
|
93
|
+
|
94
|
+
In cases where implicit searches are chained, which happens frequently
|
95
|
+
with deep UI hierarchies, if one of the searches were to fail then you
|
96
|
+
would receive a `NoMethodError` about something having
|
97
|
+
failed. Sometimes the failure would happen because the search returned
|
98
|
+
`nil`, and of course `nil` would not respond to another search, though
|
99
|
+
this problem was easy to identify if you are familiar with the
|
100
|
+
AXElements source code; in other cases the failure would occur
|
101
|
+
somewhere in the search {Accessibility::Qualifier qualifier} or in the
|
102
|
+
{Accessibility::BFEnumerator enumerator}, and it was not always clear why.
|
103
|
+
|
104
|
+
The other feature of a search failure is that the exception message
|
105
|
+
will include an element back trace using {Accessibility.path}. This is
|
106
|
+
meant to give a hint about why the search failed.
|
107
|
+
|
108
|
+
### Attribute Not Writable
|
109
|
+
|
110
|
+
You will receive {AX::Element::ReadOnlyAttribute `ReadOnlyAttribute`}
|
111
|
+
exceptions only when you try to set an attribute that is not
|
112
|
+
writable. Again, this was originally designed to more easily identify the
|
113
|
+
point of failure when you try to write to an attribute that you should
|
114
|
+
not write to.
|
115
|
+
|
116
|
+
Specifically, `set_focus` is called by methods internally by
|
117
|
+
AXElements and at one time it was causing some problems when elements
|
118
|
+
were unexpectedly not allowing their `focused` attribute to be
|
119
|
+
written.
|
120
|
+
|
121
|
+
### Attribute Not Found
|
122
|
+
|
123
|
+
A very simple fail safe that AXElements uses is the
|
124
|
+
{AX::Element::LookupFailure `LookupFailure`} exception which will be
|
125
|
+
raised when you try to explicitly access an attribute which does not
|
126
|
+
exist, or at least does not exist for the particular element that you
|
127
|
+
are trying to access.
|
128
|
+
|
129
|
+
## Not So Custom Exceptions
|
130
|
+
|
131
|
+
Sometimes it is possible that the back trace for other exceptions can
|
132
|
+
get lost. This may be a result of
|
133
|
+
[MacRuby Ticket #1369](http://www.macruby.org/trac/ticket/1369), but
|
134
|
+
it might also be because of
|
135
|
+
[MacRuby Ticket #1320](http://www.macruby.org/trac/ticket/1320) or
|
136
|
+
some other freedom patch
|
137
|
+
that AXElements or ActiveSupport adds to the run time. I have not been
|
138
|
+
able to create a reduction of the problem yet.
|
139
|
+
|
140
|
+
The real problem is that loss of back trace happens for multiple
|
141
|
+
exception classes. The [work around](https://gist.github.com/1107314)
|
142
|
+
for this case is copied to `lib/ax_elements/macruby_extensions.rb`,
|
143
|
+
but has been commented out since it causes some regression tests to
|
144
|
+
fail. If you do not get a back trace with an error then you will need
|
145
|
+
to uncomment the freedom patches or copy them to your own script.
|
146
|
+
|
147
|
+
## Disabling Compiled Code
|
148
|
+
|
149
|
+
Back traces can be lost for reasons other than a bug. When using
|
150
|
+
compiled MacRuby code, you cannot get a Ruby level back trace in case
|
151
|
+
of an error. This feature is on the road map for MacRuby, but I am not
|
152
|
+
sure when it will be done.
|
153
|
+
|
154
|
+
In the mean time, if you suspect that the portion of a back trace that
|
155
|
+
would come from a compiled file is the problem, then you can disable
|
156
|
+
loading compiled files which will force MacRuby to load source ruby
|
157
|
+
files instead. You can disable loading compiled files by setting the
|
158
|
+
`VM_DISABLE_RBO` environment variable before running a script. You can
|
159
|
+
disable loading for a single session like so:
|
160
|
+
|
161
|
+
VM_DISABLE_RBO=1 macruby my_script.rb
|
162
|
+
|
163
|
+
Other debugging options are also available from MacRuby itself. You
|
164
|
+
should check out [Hacking.rdoc](https://github.com/MacRuby/MacRuby/blob/master/HACKING.rdoc)
|
165
|
+
in the MacRuby source repository for more details.
|
166
|
+
|
167
|
+
## Logging
|
168
|
+
|
169
|
+
The core level of AXElements has logging in every case that an error
|
170
|
+
code is returned. Though, it can show false positives because of
|
171
|
+
hiccups in the accessibility API or implementation that an app
|
172
|
+
provides; so it turned off by default. This feature is also going away
|
173
|
+
in favour of more intelligent error handling in the core wrapper.
|
174
|
+
|
175
|
+
When weird bugs are occuring, possibly even crashing MacRuby, you
|
176
|
+
should try turning on the logs and then trying the script again. You
|
177
|
+
might see some tell tale logs printed out right before the crash. You
|
178
|
+
can turn on logging right after you load AXElements, like so:
|
179
|
+
|
180
|
+
require 'ax_elements'
|
181
|
+
Accessibility.log.level = Logger::DEBUG
|
182
|
+
|
183
|
+
The standard log levels are available, the full set is available
|
184
|
+
[here](http://rdoc.info/stdlib/logger/1.9.2/Logger/Severity). `Logger::DEBUG`
|
185
|
+
will turn on all logs.
|
186
|
+
|
187
|
+
## MacRuby Seems Slow
|
188
|
+
|
189
|
+
There are a few things that can cause MacRuby to be slow. At boot time
|
190
|
+
there are a number of factors, which I will cover, and at run time
|
191
|
+
there is really only one culprit.
|
192
|
+
|
193
|
+
### Long Load Times
|
194
|
+
|
195
|
+
When using certain gems, or when you have many gems installed, you
|
196
|
+
will notice that the load time for your scripts is very
|
197
|
+
long---possibly more than 10 seconds. There are many reasons why this
|
198
|
+
happens, some of which we can fix ourselves and some of which you will
|
199
|
+
have to wait for the MacRuby developers to fix.
|
200
|
+
|
201
|
+
#### Huge Literal Collections
|
202
|
+
|
203
|
+
Some gems contain source code with
|
204
|
+
[unbelievably large literal collections](https://github.com/sporkmonger/addressable/blob/master/lib/addressable/idna/pure.rb#L318),
|
205
|
+
such as the `addressable` gem. This is a problem for MacRuby for two
|
206
|
+
reasons.
|
207
|
+
|
208
|
+
First, it requires several thousand allocations at once. Most of
|
209
|
+
MacRuby's performance issues come from code that allocates too much,
|
210
|
+
and large collections can allocate several thousand objects all at
|
211
|
+
once.
|
212
|
+
|
213
|
+
The second problem is that MacRuby normally will try to JIT the code,
|
214
|
+
and the LLVM chokes on things this large. Some work has been done to
|
215
|
+
break the function up into smaller pieces (it used to take over 2
|
216
|
+
minutes to load the `addressable` gem), but it can still take a while
|
217
|
+
for MacRuby and the LLVM to work through the code
|
218
|
+
|
219
|
+
As it turns out, JIT isn't that great for short lived processes that
|
220
|
+
need to start up over and over again. In fact, JIT mode for MacRuby
|
221
|
+
was meant more for debugging.
|
222
|
+
|
223
|
+
The work around to this situation will have to come from upstream gem
|
224
|
+
developers and MacRuby itself. In the mean time, compiling these gems
|
225
|
+
will usually make them load _significantly_ faster. To compile gems,
|
226
|
+
you can install the
|
227
|
+
[`rubygems-compile`](https://github.com/ferrous26/rubygems-compile)
|
228
|
+
plug-in for rubygems. Follow the instructions from the plug-ins `README`
|
229
|
+
to learn how to use it and to know which version to install.
|
230
|
+
|
231
|
+
#### Complex Metaprogramming
|
232
|
+
|
233
|
+
Another problem that can cause long load times is complex
|
234
|
+
metaprogramming. Gems such as `rspec` do a lot of weird stuff at boot
|
235
|
+
that causes the MacRuby optimizer to do a lot of work. `rspec` alone
|
236
|
+
can add nearly 10 seconds to boot time. In this case you can tell the
|
237
|
+
optimizer to not try so hard; this will result in slower run time
|
238
|
+
performance, but it is likely worth the trade off in the case of
|
239
|
+
`rspec` (unless you compile).
|
240
|
+
|
241
|
+
You can set the optimization level for MacRuby just as you would
|
242
|
+
disable loading compiled code:
|
243
|
+
|
244
|
+
# set the level to a number between 0 and 3, 3 is the highest
|
245
|
+
VM_OPT_LEVEL=1 macruby my_script.rb
|
246
|
+
|
247
|
+
#### Large Code Bases
|
248
|
+
|
249
|
+
Large code bases taking a long time to load is not really an avoidable
|
250
|
+
situation---it happens with every project. Once again, JIT and
|
251
|
+
optimizer passes take up a lot of the load time.
|
252
|
+
|
253
|
+
In this case, it is best to compile your code (and test the compiled
|
254
|
+
version) in order to speed up boot time. You can combine compiled code
|
255
|
+
and still turn off optimization passes to get even better boot times,
|
256
|
+
but I am not sure it is worth the trade off at that point.
|
257
|
+
|
258
|
+
#### Rubygems
|
259
|
+
|
260
|
+
Rubygems suffers from a lot of technical debt. The process of
|
261
|
+
activating a gem incurs so many allocations that with as few as 25
|
262
|
+
installed gems can add an extra 2 seconds to your boot time. What is
|
263
|
+
worse, the performance degrades exponentially as you install more
|
264
|
+
gems.
|
265
|
+
|
266
|
+
The only fix for this is to cleanup and fix rubygems. Fortunately this
|
267
|
+
has been underway since the rubygems 1.4; the downside is that MacRuby
|
268
|
+
has customizations to rubygems that prevent users from upgrading
|
269
|
+
themselves. We need to wait for new MacRuby releases to bundle new
|
270
|
+
rubygems versions in order to fix this issue.
|
271
|
+
|
272
|
+
### Slow Runtime Performance
|
273
|
+
|
274
|
+
In my experience, slow runtime performance in MacRuby is almost always
|
275
|
+
the result of many allocations. If your code is runinng abnormally
|
276
|
+
slow then it is likely that you are allocating a lot of memory without
|
277
|
+
realizing it, and you should compare performance to CRuby if it is
|
278
|
+
important (and if it is possible to run the code on CRuby).
|
279
|
+
|
280
|
+
Remember that things like literal strings have to be copied every time
|
281
|
+
the line of code they are on is run, whereas immutable things like
|
282
|
+
symbols do not have to be copied. At the same time, if you pass a
|
283
|
+
symbol to a method that will coerce the symbol to a string then you
|
284
|
+
haven't saved an allocation.
|
285
|
+
|
286
|
+
Try to use in-place mutations when it is safe. An example would be
|
287
|
+
when you have to perform multiple changes to an object in a single
|
288
|
+
method, you only have to create a new copy the first time and then use
|
289
|
+
the same copy for all the other changes. Example code would look like this:
|
290
|
+
|
291
|
+
def transform string
|
292
|
+
new_string = string.gsub /pie/, 'cake'
|
293
|
+
new_string.gsub! /hate/, 'love'
|
294
|
+
new_string.upcase!
|
295
|
+
new_string
|
296
|
+
end
|
297
|
+
|
298
|
+
Remember that built-in in-place methods tend to return `nil` if they
|
299
|
+
don't make any changes, which means you need to explicitly return the
|
300
|
+
new object at the end of the method. There are still many other easily
|
301
|
+
avoidable cases where you could end up allocating a lot of memory
|
302
|
+
which are not covered. If it is important you willl just have to
|
303
|
+
analyze your code.
|
304
|
+
|
305
|
+
Sometimes allocating a lot of memory is not avoidable; running RSpec
|
306
|
+
would be an example of this. In these cases you just have to bite the
|
307
|
+
bullet.
|
308
|
+
|
309
|
+
## Don't Be Afraid To Log Bugs
|
310
|
+
|
311
|
+
Or look at the AXElements source code for that matter. The source is
|
312
|
+
well documented and hopefully not too clever, so it shouldn't be too
|
313
|
+
hard to figure things out. You can log AXElements bugs on Github where
|
314
|
+
the source is being hosted.
|
315
|
+
|
316
|
+
Though, sometimes the problem will be a MacRuby problem and the best
|
317
|
+
way to get it fixed is to
|
318
|
+
[log a bug](http://www.macruby.org/trac/). You will need to create an
|
319
|
+
account with them to log bugs.
|
320
|
+
|
321
|
+
I also recommend that you subscribe to the
|
322
|
+
[MacRuby mailing list](http://lists.macosforge.org/mailman/listinfo.cgi/macruby-devel)
|
323
|
+
to keep up to date on MacRuby developments. You can send in questions
|
324
|
+
if you are unsure about a certain behaviour, even potential bugs. It
|
325
|
+
is not a high traffic mailing list so it won't blow up your mailbox
|
326
|
+
when you subscribe.
|