Gatekeeper is Apple’s name for a feature in Mac OS X that prevents a user from opening certain programs based on a few different security preferences. (Under the hood, it’s actually part of Mac OS X’s security assessment policy subsystem, which you can manipulate from the command line with the spctl command if you have admin privileges.) If you’ve ever downloaded an app from the Internet rather than the Mac App Store, you might have seen a dialogue box appear when you tried to open the app preventing you from doing so, like this:
Ordinarily, this is (mostly) a good thing. It’s a warning from your Mac alerting you to the fact that the author of the program you’re trying to run hasn’t registered themselves with Apple. It’s not a warning that the program is actually a virus or some kind of malware—those warnings are more explicit and actually say that the program “will damage your computer”—but the fact that the developer isn’t recognized could be a legitimate concern. Of course, it might also be completely innocent, as it is in the case of TorBrowser.
If have administrator privileges on the Mac, you tell Gatekeeper to permit the program to run by right-clicking on it and selecting “Open” from the contextual menu that appears, as Apple describes in “How to open an app from an unidentified developer and exempt it from Gatekeeper.” But what if you’re not an admin user? It turns out there are a number of ways you can work around Gatekeeper to open the app anyway.
This may sound like a security risk and thus a bad idea, but there are times when you might need to do this for innocent reasons.
For example, last night, my Mac’s hard drive crashed. :( I’ve been keeping regular backups, but I’ve also been working on a programming project that’s changing rapidly. The crash happened at the worst possible time: immediately after I’d just completed another chunk of work, but before the next backup was scheduled to run. It’s only about an hour’s worth of coding, but that’s still a chunk of effort I’d like not to have lost. Naturally, I’m now running SpinRite, a hard drive maintenance and data recovery tool, on the disk, and hopefully I’ll be able to get my work back. Meanwhile, however, I’m borrowing a Mac from someone else. The account I’m using doesn’t have admin privileges, but I still need to install some programs that Gatekeeper won’t allow, such as the Tor Browser, and my favorite text editor, MacVim.
So I went about trying to find ways to work around Gatekeeper, and I found two really simple workarounds that worked for me on a fully-patched Mac OS X 10.9.5 Mavericks system with Gatekeeper enabled, of course. They may work in earlier or later versions of Mac OS X, too.
Use the command line, not the Finder
Possibly the most obvious workaround to Gatekeeper is simply to bypass it entirely by not using the Finder to open the applications that you want to run. If the program is designed as a command line application in the first place, then Gatekeeper won’t have anything to say about it. But even if it’s designed as a fully-featured graphical app, you can still launch it from the command line, thus avoiding Gatekeeper’s restrictions altogether.
For example, although Gatekeeper gets in my way after double-clicking on the TorBrowser, opening it from the command line by calling its executable directly works like a charm:
By running the app’s main executable directly, you avoid Gatekeeper in the same way as other command line applications do. Note that the open command doesn’t avoid Gatekeeper because it actually calls to the Finder.
Remove the com.apple.quarantine extended attributes
Another way to make sure that Gatekeeper doesn’t get called when you’re opening an app is to strip the app itself of the attribute that tells the Finder to call Gatekeeper. In this case, that’s the com.apple.quarantine extended file attribute. Most modern operating systems, including Mac, Linux, and Windows, have filesystems that can attach arbitrary metadata to files, and this metadata is generically known as extended file attributes. On a Mac, you can use the xattr command to inspect the extended attributes associated with any file or folder.
Indeed, inspecting the extended attributes on the TorBrowser I just downloaded revealed the relevant attribute:
$ xattr TorBrowser.app/
The -l@ switches to the more common ls command works, too, but produces different output:
$ ls -l@ TorBrowser.app/
drwxr-x---@ 6 maymay staff 204 Dec 31 1999 Contents
drwxr-x---@ 5 maymay staff 170 Dec 31 1999 TorBrowser
-rw-r-----@ 1 maymay staff 137761 Dec 31 1999 precomplete
Since we own the downloaded file, we can modify its extended attributes, no admin privileges needed. Removing com.apple.quarantine recursively is enough to disable Gatekeeper. We again use xattr, along with its -d switch, to accomplish that:
xattr -r -d com.apple.quarantine TorBrowser.app/
With the quarantine extended attribute removed, the Mac OS X Finder never calls to Gatekeeper so double-clicking on the app will work as if Gatekeeper was disabled.
So there you have it. Two simple ways to bypass Gatekeeper and open arbitrary programs even without administrator approval. None of this helps fix my broken hard drive, of course. I’ll still need to buy a replacement for my Mac (and since I work and live on a donations-only basis, if you can part with a few bucks by sending a donation to help me out here, I’d really appreciate it) but at least I can still install the tools I need to get stuff done while borrowing other laptops. And so can you!
A: In addition to the seed bank, we have an online seed exchange. It’s like a Craigslist for seeds. Members simply sign up (for free) and swap with the other 264 members. In designing the website, we worked with ‘cyber hacktivist,’ Meitar Moscovitz, to create an open-source WordPress plug-in that’s now available for free to other seed exchanges and has been downloaded 704 times worldwide.
As for “the right to be forgotten,” I strongly disagree that this is important.
First of all, no one has “the right” to be forgotten. The framing of this as a “right” is incredibly dangerous because it is by definition a way to describe other people’s memories as property that they do not actually own. I.e., when you say that someone has “the right to be forgotten,” what you are saying is that this person has ownership over other people’s memories, which sounds ridiculous right now but will actually be technologically enforceable in the near future. This “right” is thus already corrupt and extremely dystopic.
Second, even if I were to cede the ridiculous argument that this should in any way be considered a “right,” and thus bounded by the incredibly corrupt and unjust framework of State-backed Law, the fact remains that the act of “forgetting” only serves to support the abusive elements of society. The erasure of history is a defining element of oppressive practices on the macro-scale of history, but also on the micro-scale of interpersonal interaction. Men who beat their wives rely on the fact that their ex-wives do not warn their new brides that they are abusive assholes. Serial rapists rely on the fact that their victims will not have the benefit of communication with their former victims in order to perpetrate multiple assaults. The idea that we should give anyone the power to enforce a kind of “forgetting” on other people is such a terrible idea that the only reason it gains favor is either the shortsightedness or the intentional sociopathy of its proponents.
If I do something that’s embarrassing but not outright predatory, then the ethical societal reaction to this act should be compassion, not ridicule. Likewise, if I do something that is outright predatory, then it only serves my own predatory motives to be able to force other people to forget that I have done that thing. The fact that society’s reaction in the former case is not compassionate but rather abusive is a problem with society, not with the ability to keep historical records. Trying to solve society’s abusive tendencies with legally enforceable mechanisms that further protect and entrench abusive behavior is silly at best and counterproductive at worst.
However, as you can see above, they are also having some technical difficulties. Their laptop keyboard has stopped working. They’ve worked out a very temporary, clunky solution for now, in order to keep coding. But their real keyboard needs to be replaced — a repair that costs a little over $200.
Seriously, maymay has created an entire collection of tools to help and protect survivors, since the big names who run social media site can’t be bothered, and they do it completely for free and basically on their own. So yeah, if you can donate or signal boost so they could fix their keyboard, I think that’s the least we could do.
I’ve broken my laptop in the exact same way last week, and while I sympathize, I’m going to stress that this is no way gets in the way of coding. This does NOT make using any program harder. I’d love for someone to pay to fix my laptop so I can understand why they’d want someone to, especially when they provide a free service, but to say their keyboard “needs” to be replaced is kind of an overstatement, don’t you think?
Hey there. I don’t know you, so I’m assuming that means you also don’t know me and don’t know much about Maymay. But, for what it’s worth, Maymay is homeless, lives on the road, and receives absolutely zero income except for donations of money and food. And the reason they live this way is largely so that they can focus 100% on helping solve serious but not lucrative social problems that more financially-motivated others with their type of technological expertise have chosen to ignore.
Having someone else pay to replace your keyboard might be unnecessary for someone in your position, since I assume you have a job and a house. I imagine it was unpleasant but not impossible to cover the cost of your own repair. But someone who has no income, has to carry everything they own around, and does a lot of their coding in places like bus terminals and other crowded public areas doesn’t have that luxury. $200 is several months of bills (phone, server), food, and transit costs for May. They can’t realistically blow that kind of money on a computer repair.
Obviously, everybody’s got their own financial situations they’re dealing with. There’s certainly no expectation on you or anyone else to help a stranger out. Hell, I already spent all the extra money I could afford this month (and more) donating to another friend who needs help hiring a trans*-friendly custody lawyer. But for those folks who do know Maymay, understand their situation, appreciate their work, and have some extra cash to spare, making a small donation would be appreciated and very sweet.
I’m sorry, I must have been unclear to warrant this response.
“Having someone else pay to replace your keyboard might be unnecessary for someone in your position, since I assume you have a job and a house. I imagine it was unpleasant but not impossible to cover the cost of your own repair.“
There’s no way in hell I could afford a $200 laptop repair. But I’m in the exact same boat:
I don’t have to pay to fix my entire laptop, as you can see, since I bought a keyboard for $20 dollars that works just fine. No, it doesn’t look fancy. But there’s nothing wrong with it! And even though I could ask for help in getting my laptop fixed, I really feel there are better things people could be doing with $200 dollars than making my laptop look nice. $200 could buy a lot of food and blankets for a lot of people.
I think you thought I said that they should just suck it up and pay for their own laptop repairs, which is definitely not what I’m saying. I’m saying there laptop works fine, even if it doesn’t look great. I understand wanting to have a more aesthetically pleasing workplace and having the comfort of a laptop that doesn’t come in a bunch of pieces (I work from my laptop too, with probably about as much income as they receive – we’re probably very alike in the ways that we live!), but in no way does this make the laptop any less usable than a laptop with a normal keyboard.
I do suggest that, if enough donations are raised, it would be just as cost efficient to get a used or refurbished laptop altogether – unless they are using something like a mac (even then, it would be much more cost efficient to sell it and get a cheaper laptop). There are laptops in the 200-300 price range, which makes it pretty awkward to pay for repairs on a broken laptop for the same price.
I didn’t mean to cause any offense (and I certainly don’t think I meant to convey that they shouldn’t be asking for donations) but it did make me intensely curious as to how this situation could be a problem that was described as “needs fixing”.
In the very least, consider this a signal boost to my 5,765 followers.
Alright, fine, I mean this is basically splitting hairs over semantics. Certainly, nobody absolutely “needs” to have a computer that works in a specific way, or to own a computer at all, or whatever. Arguably, nobody really “needs” anything except maybe food and, even then, endless debates rage about how much and what kind of food counts as “needed” vs. “wanted.” Whatever. I’m just trying to help my friend get their laptop fixed for christs’s sake.
Here’s the thing: In my experience, the distinction between “needs” and “wants” is a pretty abritrary one. They are hardly universalizable categories, for obvious reasons. I could get into some argument here trying to justify Maymay’s decisions about why they ask for crowdfunding support around certain things (like computer repairs) and not others (for example, I personally think they could really use a warm coat that fits and some gloves, rather than just wearing three layers of beat-up hoodies full of holes and shivering all the time — but they’re never in a million years going to ask anybody else to help them with that.) But I’m not going to.
Why? Because different people have different priorities about what things make their lives more worth living. When I have a little bit of extra money, I spent it on bodywork and psychotherapy, because that’s what I feel like I “need” — and I make other decisions in my life that allow me to prioritize that. Others might argue that getting regular massage is a luxury, but they spend money on videogames or eating out at lunch or they drive a new car or own a smartphone, all things which probably feel necessary to them for their well-being (and very well might be) but that I see as indulgent expenses that I could never countenance for myself. It’s not that the way a given individual spends their resources should never be subject to analysis. It’s that these kinds of decisions are so personal and contextual, there’s no point in arguing about them at such an oversimplified level of abstraction.
The point here is that I am not Maymay’s mom, and neither are you. Repairing your laptop isn’t a priority for you, fine, don’t repair your laptop. But lecturing total strangers about how they should spend their money is just rude — whether that be lecturing Maymay about how they should purchase a more “appropriate” computer for a homeless person, or lecturing Maymay’s supporters for sending them donations instead of using that money to buy people blankets, or whatever. Come on. Nobody’s asking you to fund a financial decision you consider irresponsible. But this is the beauty of crowdfunding. If you don’t want to support the thing, don’t.
Meanwhile, there are other people for whom it feels good to have an opportunity to help out someone whose work they appreciate in a way that is directly related to the continuuance of that work. And, if there are enough of those people to raise $200, Maymay will use it to get their laptop fixed. And, if there aren’t, they’ll figure out some other way to cope. Because that’s how crowdfunding works. We’re not talking about how the government allocates our tax money, here. We’re talking about how relative strangers on Tumblr decide to spend $10.
Anyway, I’m sorry, I don’t mean to tear into you personally. I just have a lot of loved ones who are or have been homeless, and so I hear this kind of shit constantly and it drives me crazy. Homeless and poor folks are entitled to just as much agency around how they spend their income as anybody else. If you, for whatever personal reasons you have, don’t want that income to include money you gave them, don’t give them money. But suggesting that other people who do want to donate shouldn’t, because you think the person asking doesn’t really deserve the kind of support they’ve requested, is fucked up.
TBQH: My personal opinion is that, if you’d like to have a working keyboard and you can’t afford to get your laptop fixed right now, you should try asking your friends and family (or some of your 5,765 followers) for support. Why haven’t you? I bet some of them would be genuinely happy to help you out.
Thank you for the signalboost. Sincerely.
Dear Soycrates, esteemed Philosophy BA from McMaster University,
How many blankets or how much food could you have bought for homeless people with the money you spent on your tuition?
If you’re truly concerned with efficient use of money, why did you choose to major in philosophy instead of, say, business or accounting?
If you don’t want to donate money, have you considered sending me some food instead, or would you first want to know whether I eat vegan, like you and your well-cared for Vegan High Horse, or if I eat the cheaper mass-produced poison that’s actually what I can afford without stealing?
Meanwhile you think your college degree is an effective use of your money to change the world or something? Shit son, I have a bridge I’d like to sell you.
Speaking of bridges, would you like to join me for a sleepover under a highway overpass while we hide from cops? I know of a number of really good ones with relatively spacious cavities and a few even have nice views of nearby cities. Bring everything you own, including your sleeping bag, your clothes, food and water enough for three or four days, your laptop, and don’t forget your bulky external keyboard, which I’m sure you use to type out philosophical masterpieces about ethics that literally no one else on the planet can type, especially not the University Ivory Tower professors who you paid how much money to study under, again? I mean, why do either of us even bother with keyboards at all, we should just use pencils and paper, especially given that neither of us are writing computer code. Oh wait.
No but seriously, we should just use pencils. Certainly that’s a more efficient use of our money than typing code for anti-rape software that no one else is writing, anyway. But if you are gonna bring your keyboard to the sleepover, I hope the keyboard isn’t too heavy or hard to carry, because when you carry everything that you own with you wherever you go, every ounce matters, which I trust you already know because apparently our lifestyle is pretty similar, according to you and your extensive personal knowlege of my life that you acquired as a BA in Philosophy. Silly me.
Anyway, obviously you are spot-on about my main concern about my broken keyboard is how my laptop looks, because aesthetics is definitely the only thing I’ve been talking about here, unlike your morally superior focus on utility and efficiency. I’m so glad you’re here to keep me focused with your incredibly perceptive Tumblr blog!
You’ve convinced me. What I was asking for help with was stupid and entitled of me. I’m a bad, bad greedy person. So you can forget about the keyboard, and I’d be very appreciative of any donations of food you can offer instead, because then I’ll have to spend less money on food and can save more of it for other, more discretionary spending that isn’t morally approvee by esteemed ethicists like you, which of course I will check in with you about first to make sure it meets your standards of efficiency and necessity for what things my poor dirty hands touch.
But I also understand if you choose not to donate to help me out. Hell, everyone knows we homeless people are just gonna spend our money on keyboard repairs anyway. Next time when I ask for financial help, I’ll be less honest and just tell you I’m gonna buy a 40 and a dime bag, a’ight?
I’ve been enjoying my brief but focused time in The Federation and away from corporate social media so far. (“The Federation” is what we nerds who love freedom and, by extension, free software, call the distributed social network outside corporate-controlled filter bubble prisons like Facebook, Twitter, and Tumblr.) Oh sure, there’s the usual Internet asshole with a dick pic, and I expect there will be more of them as the Federation grows in number. But for now it feels like a much earlier Internet where people generally do not speak without thinking first, are reading what others write with the intention to understand instead of the intention to respond, and best of all, are posting links to tons of stuff I have never even seen before.
Partly, this is because of the inherently international flavor of the Federation. English speakers have not yet reached such numbers that English is assumed to be the default language. Half of my “timeline” is in Spanish, German, French, or some other language. And it’s an absolute joy merely to be exposed to these other languages instead of the monotonous drawl of only one of anything, be it one and only one language, one and only one news source, and so on. Plus, my Spanish and German have been getting better, so while I still need the help of a translator to engage in any sustained way, I feel myself needing it less and less to read other’s comments. Which is awesome! :D
But what’s really noteworthy about this experience is simply the vast chasm between exposure and censorship. I don’t mean censorship in the harsh sense of an iron fist where a block screen comes up as you try to access a blog post, warning you that this content is restricted by order of the government. I mean censorship in the social sense where unquestioned assumptions and complacency are left to fester like boils in our collective minds, or where outright bullying creates fascistic ideological borders that leave emotional wounds whose scars build up like walls against other people and ideas. That kind of sustained psychosomatic injury is also censorship.
But no one recognizes it as such because no one understands what censorship even means anymore, because they are being censored. After all, as the saying goes, the worst thing about censorship is [CENSORED].
I wanted to share that response here. So, here goes:
I have written about this topic a lot so I don’t actually want to repeat myself again from scratch here. However, I will summarize some of my previous thoughts and then share links to my writings elsewhere so you can evaluate them at your leisure. The main points that I feel are most important to bring up in any conversation like this one are as follows:
1. Tactics versus principles
There is a difference between a tactic and a principle. A tactic is a certain action taken for a certain purpose in a certain context. This is different from a principle, which is a general guiding philosophy used to inform a given person’s choice of what action (tactic) to take, when, and why.
Things can get confusing when the same word is used to refer to a tactic and a principle. One very common example of this is the word “violence.” There are strong and compelling arguments to be made that “violent” actions are often necessary to effect the kind of political and social changes that are desirable for all humanity, and it is in fact a form of oppression on the part of people who hold the guns to insist that all actions on the part of the people who the guns are pointed at remain non-violent. Therefore, “violence” is both a tactic (the act of militant resistance against violent killers) and also a principle (the act of forbidding a group to use violence to resist violence).
The opposite is true, too: non-violence can be used as an effective tactic (see, for instance, the Civil Rights marches, the Montgomery Bus Boycott, and so on in the US 1960’s for a few famous example), but it can also be a principle, in the form of political ideologies such as pacifism. These distinction is very important because there are situations in which a violent tactic is used as part of a non-violent principle. See, for instance, Nelson Mandela’s famously advocating the use of guns in the fight against South African Apartheid.
In your case, the question is “censorship,” but I think you are having trouble articulating your thoughts in part because you are not yet clear about the difference between censorship as a tactic versus censorship as a principle.
Second: Power always has a context
Another key point to always be aware of is the idea that “oppression” is not the same as “expression” because the former (“oppression”) always carries with it a contextual power, whereas expression does not. This makes oppression a type of expression, but it does not mean that all expressions are also forms of oppression. (This is analogous to “a square is a rectangle but a rectangle is not a square.”)
As a contrived example, imagine for a moment a child alone in their room, shouting, “I hate grown ups!” This is certainly an expression of something, but is it oppression? No, because a child alone in their room has no meaningful coercive power over adults. (That is, no significant ability to force adults to do what they demand.) The inverse, however, is not true: adults do have demonstrably strong coercive power over the child. (That is, the adult can force a child to do almost anything the adult demands.) Therefore, we say that when the child expresses hatred for adults, this is not oppression, but when we say that adults force children to do things they do not consent to doing, it is oppression, even if the adult’s actions are also well-meaning. For example, it is oppressive to force children to go to school, even though the adult who forces their child to go to school believes such force is in the best interest of the child. Another, less controversial example: it is oppressive to hit children in order to make them behave as instructed, even if the adult believes such force to be in the best interests of the child. Both forceful acts are expressions of oppression, by definition.
The reason this contextualization matters so much is because it explains why “freedom of oppression” is not a meaningful statement (it is an oxymoron) but “freedom of expression” is meaningful, and important. This also explains why there are so many disagreements about what the difference between “oppression” and “expression” actually is in a given context, and I argue that this confusion is largely an intentional effort on the part of oppressors (i.e., on the part of people in power) to muddy the waters by ignoring or discarding important historical, individual, and other contexts. In fact, if you look at history, you will often see groups who take power violently also simultaneously try to destroy historical artifacts, records, and other evidence of a given context. This is not an accident, they know exactly what they are doing: they are creating an environment in which they can control and define what the context is. If they succeed, they can argue that something which was empirically not true is true, such as parents who argue that “my child can force me to do all sorts of stuff!” This is just nonsense, but if people believe it, it does not matter that it is not true, because they collectively act as if it is true. That collective delusion is therefore also an oppressive expression.
This action of rewriting history is called “erasure,” and if you look carefully for it, you will find it all over the Internet, especially (because digital conversations are so easy to erase, and censor). Erasure is a classic pattern of bullying. What you should watch out for is when Person A says something that causes hurt to Person B, and then Person B says something that hurts person A. Person A then typically responds as if the start of the conversation was when Person B said the thing that insulted them, and disregards the fact that Person B was responding to something that they, themselves, said or did that was hurtful.
So, actually, it really matters “who started it,” because that history is part of the context that informs the judgement about whether an act was oppressive or merely expressive.
Third: Words are defined by people’s reaction to them, not by the dictionary
Finally, it’s also important to realize that the single most powerful weapon that oppression has at its disposal, the tool every oppressive act must use in order to sustain itself, is the re-definition of a word or idea to mean something that it originally did not. This is technically called “appropriation” or, sometimes, “assimilation,” and it is a very nasty thing because it is so hard to untangle after it has happened. This can also be understood as a form of erasure (see above), because the point is to re-write the history of how the word or idea came to be in such a way as to make the original meaning or context hard to know.
One very common example of this is the word “censorship” itself. You are using “censorship” to mean a specific thing: the silencing of some voices and opinions in a given space. But that is not the original meaning of censorship at all. Your first clue that this was not the original meaning of censorship is that it is a different word than silencing. ;) Why have two words that mean the exact same thing, if there is not some important distinction or motivation to create the second word?
Censorship actually means “the active suppression of points of view to such a degree that those points of view do not have the ability to influence anyone who might, nevertheless, encounter them.” This is very different from silencing. It is very different from moderating comments on forums. It is a definition that acknowledges the ways in which censorship can be more than just pressing the “delete” button on a comment. For example, by this definition, censorship can also include purposefully slowing down Internet connections (“bandwidth throttling” is a form of censorship).
What’s important to realize here is that people who want to retain power they already have almost always use that power to redefine and narrow the definitions of words that were originally used to resist that power. Another good example of this is “consent,” which many people now treat as “the same thing as” the word “permission.” But these two words are not the same and, again, your first clue that they are not the same should be that they are two different words. Consent is not permission, and permission is not consent, but you can rest assured that most people (especially men) will tell you that they are, in fact, the same.
Okay, that was actually a lot longer than I had intended to write, but it was still just a summary of my many other writings on the topic. You can find a lot of my further writings about this topic by accessing the following websites, and then following links to even more of my writings if they are of interest to you. Some good places to start are:
There are actually a lot more posts than just these. For instance, you already read my “Complicity with Abuse: 101-level information social justice hobbyists are dangerously ignorant of” essay, which has a similar theme to this post. My point is simply that most of the people who are talking about these issues do not do so with much education about the topic, nor with any genuine interest to actually acquire any deep knowledge of it. The ones that do desire this are often frustrated by attempts to prevent them from gaining that knowledge (they are censored), but because these self-motivated learners are starting from a disadvantage, they inevitably believe some of the propaganda and lies that the censors supply. The most dangerous and insidious of these beliefs is that censorship (or oppression) itself is limited only to what the censors or oppressors themselves define as censorship or oppression.
I’m sure you can understand why that is a very clever way to enact censorship. ;)
Let me repeat that: while I am still “on Tumblr” and so on for now, my archives will not remain available for very long. If you find something of mine useful, you will need to make a copy of it and host it yourself.
The errors you see when you just punch in my web address in your browser or follow a link from Google are not happening because my blogs “broke.” The errors are intentional; my blogs have simply become invisible to some while still being easily accessible to others. […] Think of my web presence like Harry Potter’s Diagon Alley; so hidden from Muggles that they don’t even know what they’re missing, but if you know which brick to tap, a whole world of exciting new things awaits you….
As a result, a number of you have already asked the logical question: “Is there some easy way to automatically download your archives, instead of manually copy-and-pasting almost a decade of your posts? That would take forever!”
The answer, of course, is yes. This post is a short tutorial that I hope gives you the knowledge you need to download an entire website for offline viewing. This will work for any simple website like most blogs and personal sites, including mine. Archival geeks, this one’s for you. ;)
A sculptor must understand stone: Know thy materials
A website is just a bunch of files. On a server, it usually looks exactly like your own computer’s desktop. A page is a file. A slash (/) indicates a folder.
Let’s say you have a website called “my-blog.com.” When you go to this website in a Web browser, the address bar says: http://my-blog.com/ What that address bar is saying, in oversimplified English, is something like, “Hey, Web browser, connect to the computer at my-blog.com and open the first file in the first folder you find for me.” That file is usually the home page. On a blog, this is usually the list of recent posts.
Then, to continue the example, let’s say you click on a blog post’s title, which is a link to a page that only contains that one blog post. This is often called a “permalink.” When the page loads, the address bar changes to something like http://my-blog.com/posts/123456. Again, in oversimplified English, what the address bar is saying is something like, “Hey, Web browser, make another connection to the computer at my-blog.com and open up the file called 123456 inside that computer’s posts folder.”
And that’s how Web browsing works, in a nutshell. Since websites are just files inside folders, the same basic rules apply to webpages as the ones that apply to files and folders on your own laptop. To save a file, you give it a name, and put it a folder. When you move a file from one folder to another, it stops being available at the old location and becomes available at the new location. You can copy a file from one folder as a new file in another folder, and now you have two copies of that file.
In the case of the web, a “file” is just a “page,” so “copying webpages” is the exact same thing as “copying files.”
Now, as many of you already surmised, you could manually go to a website, open the File menu in your Web browser, choose the Save option, give the file a name, put it in a folder, then click the link to the first entry on the web page to load that post, open the File menu in your Web browser, choose the Save option, give the file another name, put it in a folder, and so on and so on until your eyes bled and you went insane from treating yourself in the same dehumanizing way your bosses already treat you at work. Or you could realize that doing the same basic operation many times in quick succession is what computers were invented to do, and you could automate the process of downloading websites like this by using a software program (a tool) designed to do exactly that.
It just so happens that this kind of task is so common that there are dozens of software programs that do exactly this thing.
A sculptor must understand a chisel: Know thy toolbox
I’m not going to go through the many dozens if not hundreds of tools available to automatically download things from the Web. There is almost certainly an “auto-downloader” plugin available for your favorite Web browser. Feel free to find one and give it a try. Instead, I’m going to walk you through how to use simply the best, most efficient, and most powerful of these tools. It’s called wget. It stands for “Web get” and, as the name implies, it “gets stuff from the Web.”
If you’re not on Windows, or if you just want to learn how to use wget to copy a website directly, then read on. You may also want to read on to learn more about the relevant options you can enable in wget so it works even under the most hostile conditions (like a flaky Wi-Fi connection).
Relevant wget options
While there are dozens upon dozens of wget options to the point that I know of no one who has read the entire wget manual from front to back, there are only three options that really matter for our purposes. These are:
-m or --mirror
This options turns on options suitable for mirroring. In other words, with this option enabled, wget will look at the URL you gave it, and then copy the page at that URL and all pages that first page links to which also start with the same URL as the URL of the first page until there are no more links to follow. How handy! ;)
-k or --convert-links
The manual describes this option better than I could. It reads:
After the download is complete, convert the links in the document to make them suitable for local viewing. This affects not only the visible hyperlinks, but any part of the document that links to external content, such as embedded images, links to style sheets, hyperlinks to non-HTML content, etc.
So in other words, after the download finishes, all links that originally pointed to “the computer at my-blog.com” will now point to the archived copy of the file wget downloaded for you, so you can click links in your archived copy and they will work just as they did on the original site. Woot!
This option isn’t strictly necessary, but if you’re on a flaky Wi-Fi network or the server hosting the website you’re trying to download is itself kind of flaky (that is, maybe it goes down every once in a while and you don’t always know when that will be), then adding this option makes wget keep trying to download the pages you’ve told it are there even if it’s not able to make a connection to the website. Basically, this option makes wget totally trust you when you tell it to go download some stuff, even if it tries to do that and isn’t able to get it when it tries to. I strongly suggest using this option to get archives of my sites.
Okay, with that necessary background explained, let’s move on to actually using wget to copy whole websites.
Preparation: Get wget if you don’t already have it
Step 1: Make a new folder to keep all the stuff you’re about to download
This is easy. Just make a new folder to keep all the pages you’re going to copy. Yup, that’s it. :)
Step 2: Run wget with its mirroring options enabled
Now that we have a place to keep all the stuff we’re about to download, we need to let wget do its work for us. So, first, go to the folder you made. If you’ve made a folder called “Mirror of my-blog.com” on your Desktop, then you can go into that folder by typing cd "~/Desktop/Mirror of my-blog.com" at a command prompt.
Windows users will have to dig around the WinWGet options panes and make sure the “mirror” and “convert-links” checkboxes are enabled, rather than just typing those options out on the command line. Obviously, replace http://my-blog.com/ with whatever website you want to copy. For instance, replace it with http://days.maybemaimed.com/ to download everything I’ve ever posted to my Tumblr blog. You’ll immediately see a lot of output from your terminal that looks like this:
Now just sit back, relax, let wget work for as long as it needs to (which could take hours, depending on the quality of your Internet connection). Meanwhile, rejoice in the knowledge that you never need to treat yourself like a piece of dehumanized machinery ever again because, y’know, we actually have machines for that.
Even before wget finishes its work, though, you’ll see files start appearing inside the folder you made. You can now drag-and-drop one of those files into your Web browser window to open that file. It will look exactly like the blog web page from which it was downloaded. Voila! Archive successfully made!
Special secret bonuses
The above easily works on any publicly accessible website. These are websites that you don’t need to log into to see. But you can also do the same thing on websites that do require you to log into them, though I’ll leave that as an exercise for the reader. All you have to do is learn a few different wget options, which are all explained in the wget manual. (Hint: The option you want to read up on is the --load-cookies option.)
What I do want to explain, however, is that the above procedure won’t currently work on some of my other blogs because of additional techno-trickery I’m doing to keep the Muggles out, as I mentioned at the start of this post. However, I’ve already created an archive copy of my other (non-Tumblr) sites, so you don’t have to.1 Still, though, if you can figure out which bricks to tap, you can still create your own archive of my proverbial Diagon Alley.
Finally, if data archival and preservation is something that really spins your propeller and you don’t already know about it, consider browsing on over to The Internet Archive at Archive.org. If you live in San Francisco, they offer free lunches to the public every Friday (which are FUCKING CATERED AND DELICIOUS, I’VE BEEN), and they always have need of volunteers.
If you’re just curious, the archive contains every conference presentation I’ve ever given, including video recordings, presentation slides, and so on, as well as audio files of some podcasts and interviews I’ve given, transcripts of every one of these, all pictures uploaded to my site, etc., and weighs in at approximately 1 gigabyte, uncompressed. [↩]
retract (remove) posts from Diaspora when they are deleted from your WordPress blog
completely customize or omit the footer of your Diaspora post
crosspost custom post types or even write another plugin to programmatically change how Diasposter formats your post.
Diasposter was inspired by the great WP to Diaspora plugin, which I really wanted to make use of but didn’t feel totally comfortable with because of how it stored Diaspora session information in relatively insecure file on the filesystem. In contrast, Diasposter never writes your Diaspora login cookie to disk so your D* access credentials are kept as safe as you make your WordPress database.
When I first installed this plugin, there was an automatic horizontal scrollbar so that users could move to see all of the columns. However, it has now disappeared which means one of the columns is not fully readable.
Can you help?
Right above the button you clicked to post this question there is a line of text that reads:
Did you include a link to your site, so that others can see the problem?
Given that you didn’t notice this, I am going to suggest that you slow down and think about what was different on your site from when you installed the plugin (and experienced it working as expected) and now (when it’s not). If you still need help after that, I suggest you first think more about the answer to the question quoted above before you post again.
I apologise, it was an oversight on my behalf, as you have pointed out. Put it down to Friday ‘end of the week’ fuzzy head, if you like.
The pages where we are currently using the plugin are [here and here].
To clarify, if you hover around the rows and columns, it appears you can swipe and move it around, but the visible arrows and scrollbar is not visible. We have a lot of not very IT-literate people who use our website for support so it would be handy to make it like a visible scrollbar to click on again, if possible.
Thanks for your patience.
A clarifying question: you want a scrollbar to appear but one does not exist even when the browser window is too narrow to fit the whole table?
That is to say, I am confused by the statement “the visible arrows and scrollbar is not visible.” :/
Yes, we want a scrollbar and the arrows that go either end of said scrollbar to display because the browser window is too narrow to fit the whole table.
We feel that people would prefer to have the arrows on the scrollbar visible to encourage them to click them so that they can see the columns that go off of the screen.
So, when I go to your pages they both show scroll bars just as you say you want them. :\ I’m afraid whatever you’re seeing is specific to your combination of computer and browser.
It’s very likely that users who browse to your site are seeing the scroll bars show automatically. At least, that’s what happens when I load the site.
The beauty of the Web is that users are able to define their experience so it suits them best. Users whose browsers and operating systems are set to show scroll bars are showing scroll bars. This is good news, because it means you do not need to worry about your site malfunctioning: it is not malfunctioning and, as you say, nothing about it has changed.
Note: This guide assumes you never used BitTorrent before, and that you want to start learning about it with a safety focus from the outset, but it does assume you understand basic computer and Web lingo like “website address” and “downloading.” If you’re new to BitTorrent and don’t care about staying private, then LifeHacker’s “A beginner’s guide to BitTorrent” or “The Torrent Guide for Everyone” at MakeUseOf.com may be more your speed. Also, if you do have some experience with BitTorrent, all the better. Things will make more sense to you more quickly. :)
If you’re going to read this how-to guide, I’m not going to assume you need to be convinced that downloading movies and other digital media like music, eBooks, games, and so on is something worth doing. There are, of course, many reasons why you might want to get media at no cost. These reasons range from the personal (Netflix doesn’t have the show you’re really into right now) to the political (fuck Netflix and also fuck capitalism) and everything in between (you don’t have “discretionary income” because, y’know, capitalism, but whatevs).
This guide isn’t trying to tell you what you should do—that’s your government’s job. All I want this guide to do is help you access the material you want, whatever that material is, regardless of why you want it, safely and anonymously. And since the most widely used and arguably most effective digital media distribution technology is BitTorrent, that’s what we’ll focus on today.
If you’ve heard anything about “downloading free movies on the Internet,” you probably heard of BitTorrent or its more colloquial synonym, “torrents.” You have also probably heard of companies threatening BitTorrent users with Internet service bans, financial penalties, and even lawsuits for “stealing intellectual property.” Through expensive and coordinated campaigns, companies like Disney and others represented by special interest groups like the RIAA and MPAA try to convince people that BitTorrent is hard, immoral, and unsafe to use. But these corporate-backed efforts are little more than self-serving moral crusades, effectively a big societal guilt trip, and a false one at that.
By the time you finish reading this guide, you’ll see how and why BitTorrent is easy, ethical, and safe to use. To do that, let’s start at the beginning. (If you’re one of those “just give me a fish, I don’t want to learn how to fish,” people, skip to Step 1, below. But I warn you, you’ll have a much better understanding of what I’m talking about, and that means you’ll be able to keep yourself a lot safer, if you read thoroughly than if you skip ahead. You can always skip ahead the next time, after you absorb the background information first.)
What is BitTorrent, really?
BitTorrent is a way to copy files between computers. That’s it. Really. “But if it’s just a way to copy files around,” you’re probably wondering, “what makes BitTorrent so special?”
What makes BitTorrent special: pieces, not files
What makes BitTorrent special is the way it goes about copying (or “sharing”) files. Ordinarily, to start downloading a copy of a file from someone else, they have to have the entire file. Not so with BitTorrent. Using BitTorrent, you can download (that is, receive, or copy)1incomplete parts of a file from someone else who also only has some but not all parts of the desired file, themselves. Moreover, BitTorrent itself doesn’t care what the file is. The file might be a “pirated” movie, but it might also be literally anything else. BitTorrent isn’t just for piracy. BitTorrent can be, and often is, used to share anything that can be digitized, no holds barred, since all BitTorrent cares about are what it calls pieces.
You can think of BitTorrent “pieces” like pieces in a jigsaw puzzle. When you download a file with BitTorrent, what you’re actually downloading is copies of all the individual puzzle pieces that, taken together, make up the completed jigsaw puzzle. When you have all the pieces, BitTorrent automatically puts the pieces in the right places so they make up the desired file or files. This completed puzzle is what BitTorrent calls a “seed.”2
When you first go to download stuff with BitTorrent, you’ll be presented with something called a “torrent.” You can think of torrents, which are sometimes a kind of file themselves (a “.torrent file”), like empty jigsaw puzzle boxes.3 Torrent files describe their contents, but they are not the actual content. So the torrent isn’t a true puzzle piece, per se. Rather, a torrent is the additional information we need to look for the rest of the pieces. Much as a picture on a jigsaw puzzle box shows you what the complete puzzle with all the pieces in the right spots is supposed to look like, but it isn’t a puzzle piece itself, so too does the torrent itself describe the completed torrent contents.
Okay, but where do torrents come from?
Torrents are made by other people, just like you. Sharing something of your own with others using BitTorrent is actually pretty easy. Beyond that, especially if you want to share something big but don’t have a lot of resources like bandwidth or disk space, using BitTorrent to share it can help you out big time.
Making a torrent is a simple matter of using a program to make a new .torrent file (our proverbial “jigsaw puzzle box”) out of files you already have on your computer. Usually, this is as simple as choosing “New Torrent…” from the “File” menu in a BitTorrent app, such as in this screenshot of uTorrent 1.8.4 for Mac OS X, below:
There are a lot of BitTorrent client apps you can choose to use. Naturally, Wikipedia has a huge comparison chart of them, and plenty of blogs have their “best of” picks. “uTorrent” is just the name of a popular one, and I like it well enough. For our purposes, the only really important thing is that whatever client you pick needs to have support for network proxies, which we’ll talk about next. If you’re not sure what to use, I recommend Deluge because it’s free, fast, works everywhere, and yes, supports proxies.
Sharing something of your own makes you the first seed for this torrent, since you obviously have all the pieces of the files you’re sharing. For other interested users to make their own complete copies of a file, there needs to be at least one person who’s got all the puzzle pieces. That is, there needs to be at least one “seeder.” But don’t worry, because once someone else finishes downloading all the pieces of your torrent from you, they become another seeder, since now they, too, have all the puzzle pieces. The group of folks sharing a torrent is what BitTorrent calls a “swarm.”
Any torrent you might download was first uploaded to the Internet by someone else in this way. Many people also take the next step of listing their torrents in one or more of the many public, searchable directories that index, archive, and categorize torrents, called “torrent sites.” The most famous of these community-driven websites was ThePirateBay.com (may it rest in peace), but today there are dozens of popular ones, like Kickass Torrents, ExtraTorrent, and AhaShare, to name a few.
So you see, there’s nothing nefarious, underhanded, or dangerous about sharing files over BitTorrent, or “torrenting.” The danger comes solely from malicious, overbearing, and greedy people who abuse others to try to control what is being shared, and who is allowed to share it. BitTorrent itself is just a tool, much like the rest of the Internet. Equating BitTorrent with wrongdoing is like telling people they’re doing something wrong when they browse the Web and read blogs. It’s just silly. Still, much like browsing the Internet, torrenting can be dangerous because of the malicious people, corporations, or tentacle monsters who are willing and able to abuse you for reading, saying, or sharing something they don’t want you to.
That’s where protecting yourself with public proxies comes in.
Public proxies as shields for our identity
To keep yourself safe from predators like corporations with itchy lawsuit fingers, you have to know a thing or two about network proxy servers.
Everything you need to know about proxies, and nothing you don’t
A network proxy is simply a computer that’s willing to forward a message from you to another computer on your behalf, just like a classmate who’s willing to pass your note along to a fellow classmate during class.
The nature of proxies makes them easy to abuse (much like a classmate’s trust), so many proxies are not available for public use. These are called “private proxies,” and they generally require that you have a username and a password before you can use them. But many other proxies are available for public use, offered freely to netizens like you by folks who understand the importance of a free and open Internet. Many of these proxies are listed in directories like XRoxy.com, ProxyNova.com, UltraProxies.com, PublicProxyServers.com, and so on. These are called “public proxies,” to no one’s surprise.
By the same token, an anonymous proxy is a computer that’s willing to forward a message from you to another computer, and won’t tell that other computer who the message came from. For obvious reasons, anonymous network proxies are almost always also public proxies. Anonymous proxies are designed to shield our identities. The ones that take privacy seriously are like our best friends; they’re willing to pass notes for us in class and won’t snitch on us if they get in trouble.
Confusingly, there are about as many ways to describe (or “classify”) anonymous proxies as there are websites listing them. Each of the proxy directories I just mentioned displays their list somewhat differently, but they all classify proxies along the same basic criteria because, technically, they’re all doing the same thing. For our purposes, the important things to know about a proxy are:5
Its IP address and port number. This is the proxy’s internet address. It serves the same purpose as the Web addresses (URLs) you’re familiar with, but both parts are only numbers.
Its anonymizing features. We only want to use proxies that are labelled with words like “HiAnon,” “High KA+,” “Elite,” “Anonymous,” or “Ultra” anonymity levels. We don’t want “transparent” proxies.
Ideally, the proxy should also have “SSL,” “TLS,” or “HTTPS” support, which all mean the same thing. These proxies are the ones that accept the ubiquitous, encrypted Web traffic that many banks and e-commerce sites like Amazon and Facebook use. This is helpful to us because it means we can make our torrenting activity look like we’re browsing websites, even if we’re not.
Finally, there’s one peculiar and popular kind of anonymizing proxy that deserves a special mention: the Tor Browser.
Tor Browser: a very special, very important anonymizing tool
That said, Tor does come in handy when you’re searching the Internet for torrents to download in the first place. Especially if you’re looking for some “intellectual property” protected by the legal system (*cough*movies/games/TV shows/apps/etc.*cough*), you probably want to use the Tor Browser to find torrents of it. This is because, if you use your regular ol’ Web browser and Internet connection directly, you’ll reveal what you’re searching for to anyone looking. Using the Tor Browser, rather than your regular browser, keeps others in the dark. Tor, in turn, then makes a request to The Pirate Bay or whoever on your behalf, and returns their answer (that is, the resulting web page) to you. So as long as you use the Tor Browser, your Internet service provider (and your government) remains none the wiser about what material you’re accessing.
This is how we want our actual torrenting to work, too. So what we need is a (fast) network proxy. But wait. We don’t want anyone to know that we’re looking for fast network proxies, because duh. What are we to do? The answer, by now, should be obvious: use Tor. So, if you don’t already have it, get the Tor Browser. It’s an easy to use Web browser that frees you to browse the Internet anonymously. We’ll need that.6
Torrenting anonymously: an overview
At this point, you have all the knowledge you need to torrent anonymously, even if you don’t feel like it, yet. Let’s review.
Torrenting is simply the activity of copying files from one computer to another using BitTorrent. To do this anonymously, you need to make sure that you never make a direct connection from your computer to other users. For that, you use a BitTorrent client that supports network proxies. You also need a way to search for torrents that your Internet provider won’t know about. For that, you use the Tor Browser.
You also use the Tor Browser to find an anonymous proxy. Once you find an anonymous proxy, you tell your BitTorrent client to use it, which is a simple matter of setting its proxy preferences to the address of the anonymous proxy you found. Then you load the torrent you want to download into your BitTorrent client, and you’re done.
That’s the whole process from start to finish. Now let’s take each step one at a time.
Preparation: Get the Tor Browser and a BitTorrent client that supports proxies
If you don’t already have it, download and open the Tor Browser. If you don’t already have a favorite BitTorrent client that supports proxies, try Deluge. Open it up, too.
Step 1: Find and configure an anonymous network proxy
As mentioned earlier, there isn’t anything magical about network proxies, and there’s nothing special you need to know to use one. All you really need to know about them is where they are. Luckily for us, there are public listings of them in much the same way that there are public listings of torrents themselves. A simple search for them on the Tor Browser’s home page, as shown here, will return many such listings:
Make a note of the type (typically either SOCKS4, SOCKS5, HTTP, or HTTPS) and address (IP and port numbers) of an anonymous proxy. Then, configure your BitTorrent client with those network settings.
Here’s an example of what Deluge 1.3.11’s Proxy preference window looks like:
And here’s the same configuration in uTorrent 1.8.4 for Mac OS X:
Anonymous proxies don’t stay online forever. They come and go at irregular, sometimes unexpected intervals. That’s why they’re listed in public directories alongside a timestamp, so you know when the proxy was last checked to be working.
Whenever you start torrenting, check the proxy settings in your BitTorrent client to make sure you’re using a proxy that is currently online.
Step 1-A: Enable BitTorrent encryption settings
While we’re digging around BitTorrent client preferences, it’s worth taking a look at some other settings related to privacy. Unlike the earlier settings, which you’ll need to adjust with a new anonymous proxy every time you want to start downloading a torrent, these are all “set it and forget it” options, and they all do basically one thing: turn on BitTorrent’s protocol encryption.
In uTorrent, you’ll find the protocol encryption options in the “BitTorrent” preference pane, where you want to set the “Outgoing encryption” option to “Force,” as shown here:
In Deluge, you’ll find the same options in the “Network” preference pane, grouped under the “Encryption” header:
Set all the encryption options available to you to “Force,” meaning that you’ll only accept encrypted connections. These settings help hide that the messages you’re sending through the proxy are BitTorrent messages, which means proxy operators and Internet service providers who don’t like torrent traffic will be less likely to notice that you’re torrenting. This also might mean you reject connections from some BitTorrent users who don’t support encryption, but these days enough folks do that you probably won’t notice a difference.
In my experience, using BitTorrent’s protocol encryption settings and getting into the habit of choosing a new anonymous proxy each time you start a torrenting session means you can fly under the radar and still use BitTorrent for a good, long while. Permissive coffee shops or other free Wi-Fi spots, instead of your home connection, are also good spots to torrent from. Just make sure you’re still torrenting using encryption and a working anonymous proxy—and that you tipped your barista. (Use cash.)
Step 2: Find a torrent to download
Back in the Tor Browser, click around some of the torrent sites or use their search features to find a torrent you want to download. Here’s what a search for “daily show” looks like at Kickass Torrents:
As with most things in life, use common sense to help guide you to a good torrent. Remember, these torrent descriptions are like the outside of a product box; they’re labels, not the actual contents. Here are some common sense questions you could ask yourself to avoid fakes and scams as you browse for a good torrent:
Does the reported file size of the torrent seem reasonable for what you’re trying to download?
Does the torrent’s reported file count and content list seem reasonable for what you’re downloading?
Skim the comments associated with the torrent listing. Do commenters tend to agree that the torrent is high quality, or do they call it a fake?
Does the torrent have a healthy number of seeds? Remember, if a torrent has no seeds at all, then you’re not going to be able to complete your download.8
Step 3: Download the torrent file or click the magnet link, and load the torrent in your BitTorrent app
Once you find a suitable torrent, find and click the download link on the torrent site (usually a downwards-pointing arrow), or the magnet link (usually a horseshoe magnet icon). One, if not both, of these options will cause your browser to pass along the data to your BitTorrent app, which may ask you where you want to save the torrent contents. Choose an appropriate spot on your hard drive and start torrenting.
If you’ve set everything up by following the previous sections, you’ll begin to see connections appear in your BitTorrent client. In actuality, though, none will be direct connections you’ve made to any of those machines. Instead, those connections are being routed through the anonymous proxy you chose, earlier.
Congrats, matey! ;) You can now cancel your Hulu subscription and enjoy the wonderful world of media without commercial pollution interruption.
Optionally, verify it’s all working as intended
The simplest way to verify your setup is to use the netstat utility built in to your computer. This is a command line tool that shows you the state of all network connections your computer is currently making. On a Mac as well as most Linuxes and other UNIX-like systems, the easiest way to get the output we want is to invoke the netstat program with its -n and -p switches enabled. Open a new Terminal window, type netstat -np tcp, and then press return.9 This will produce a report looking something like the following:
$ netstat -np tcp
Active Internet connections
Proto Recv-Q Send-Q Local Address Foreign Address (state)
tcp4 0 0 192.168.1.77.54535 220.127.116.11.8080 ESTABLISHED
tcp4 0 0 192.168.1.77.54534 18.104.22.168.8080 ESTABLISHED
tcp4 0 0 192.168.1.77.54533 22.214.171.124.8080 ESTABLISHED
tcp4 0 0 192.168.1.77.54532 126.96.36.199.8080 ESTABLISHED
tcp4 0 0 192.168.1.77.54531 188.8.131.52.8080 ESTABLISHED
tcp4 0 0 192.168.1.77.54530 184.108.40.206.8080 ESTABLISHED
tcp4 0 0 192.168.1.77.54529 220.127.116.11.8080 ESTABLISHED
tcp4 0 0 127.0.0.1.9151 127.0.0.1.53404 ESTABLISHED
tcp4 0 0 127.0.0.1.53404 127.0.0.1.9151 ESTABLISHED
tcp4 0 0 192.168.1.77.53403 18.104.22.168.9001 ESTABLISHED
tcp4 0 0 127.0.0.1.9151 127.0.0.1.53402 ESTABLISHED
tcp4 0 0 127.0.0.1.53402 127.0.0.1.9151 ESTABLISHED
tcp4 0 0 192.168.1.77.53395 22.214.171.124.5223 ESTABLISHED
tcp4 0 0 127.0.0.1.49153 127.0.0.1.1023 ESTABLISHED
tcp4 0 0 127.0.0.1.1023 127.0.0.1.49153 ESTABLISHED
The important column for our purposes is the fifth one, “Foreign Address,” which lists the addresses of computers our computer is directly connected to. If you see the address of the proxy server you chose to use back in step 1, congrats, you’re torrenting via a proxy. What you don’t want to see there is the address of any peers (fellow participants in the swarm) that your BitTorrent client shows you:
If you see these addresses in netstat‘s “Foreign Address” column, it means your computer is directly connecting to the swarm, without using a proxy. As shown in the example above, comparing netstat‘s output with the peer list here shows that we’re set up nicely!
Torrenting anonymously can be tricky, and it certainly helps to have a friendly neighborhood technomage to ask for advice if you’re feeling uneasy. But as you can see, it isn’t a hard thing to learn once you have the information you need. Best of all, I can guarantee you this: torrent sites have a far better selection than Netflix. ;)
The best source for information about news and developments in the BitTorrent universe is TorrentFreak.com. Check it out!
Addendum: What about paid VPN services?
A link to this guide was posted on The Pirate Bay subreddit, where it was well-received. Several people suggested using paid VPN services instead of free public proxies. Using a paid VPN service means purchasing access to a private proxy from a corporate entity, which is something I do not recommend. There were also some good follow-up questions about using proxies. Here’s one such exchange.
The first proxy I tried didn’t work (cmd was showing peer addresses), I tried another and it worked.
However, the first time, the torrent still downloaded, I wouldn’t have known it wasn’t going through the proxy, so is it necessary to check the connections every time I use a new proxy, start utorrent etc? What if the proxy stops working halfway through? It would be nice for utorrent to stop downloading if it’s not going through the proxy.
Also, why are paid VPNs so recommended if it’s as easy (and cheap) as this to avoid punishment?
Yes, you should always check to see if the proxy you selected is actually working for each new proxy you select, and you should always choose a new proxy each time you fire up your torrent client. People pay money for reliability, not technology. The technology of a free proxy and the technology of a paid proxy is identical. The difference is the consistency and contractual uptime guarantee, that’s it. If you have bad habits (i.e., you’re not carefully checking to see if the settings you entered ACTUALLY WORK after you enter them), then maybe paying for reliability is something you want to do. Just keep in mind that when you pay for something, there’s a money trail. If you don’t use a paid service, there’s no receipt in someone’s financial accounting ledger that can be traced back to you. I prefer being careful and using free proxies over paying money for the same technology in a way that encourages me to be lazy.
Also, uTorrent has a bad habit of trying proxies you give it but falling back to whatever other means are available if the proxy doesn’t respond to you. That’s a problem with uTorrent, not you. It does mean that you might want to use a different BitTorrent client, though, if you’re not careful about checking the proxies you choose to use. Again, that’s why I like Deluge: if you give it a proxy that isn’t alive, it will try the proxy but the download will fail. That’s what you want, because the proxy is dead. You don’t want it switching to a direct connection behind your back.
Another thing you can do to be extra cautious is disable PEX (Peer Exchange) and DHT (Distributed Hash Table) lookups. Some clients don’t respect proxy settings for these two mechanisms, even though they should. Again, that’s a problem with the client, not you. It’s also easier to check that the proxy you’re setting up is really the proxy being used for trackers (as opposed to PEX or DHT) because numerous paid torrent VPN services also provide IP checking utilities that you can by definition use for free (even without being a customer). See, for instance, BTGuard: CheckMyTorrentIP. Just set up a free anonymous proxy like in the above HowTo guide, then follow the steps in the BTGuard CheckMyTorrentIP page, and look for the IP address of the proxy you set up instead of “BTGUARD IS WORKING”. BTGuard won’t tell you “it’s working” because you’re not paying or using BTGuard. But it will show you what IP address it thinks you’re using, and if that IP address is the address of the proxy you configured, then you’re torrenting (at least using traditional trackers) through that proxy. :)
Hope this helps.
Everyone glosses over this point, myself included, but this point is actually very important: downloading something is the act of making a copy of the thing you are downloading. When you download or receive something from the Internet, you are literally creating an exact replica (a “copy”) of whatever it is you’re downloading. This is fundamentally different from “stealing,” which is the act of removing something from one place and putting it in another. You literally can not steal anything using the Internet no matter what you do, and anyone who tells you differently is trying to sell you something, or put you in jail. [↩]
BitTorrent files typically have the .torrent file name extension, but they don’t have to. Technically, BitTorrent files are bencoded dictionaries, which is a fancy way of saying an index that lists the torrent’s referenced contents and where that content can be found. So they’re really meta or metainfo files. Sometimes they’re also called a manifest. When torrents are not files, they are usually something called “magnet links,” which serve the same purpose as a torrent file’s dictionary but that use a special URL instead of an actual file. More on magnet links later. [↩]
In the screenshot above, you might have noticed there’s a field for “Trackers.” That’s where you paste the URLs (the Web addresses) of any well-known trackers you want to announce yourself to. When you start sharing (or “seeding”) your file, those trackers will dutifully re-announce your announcement of your presence to any other BitTorrent users who want to copy what you’re sharing. There are numerous stable, public, free BitTorrent trackers available to you (like udp://tracker.publicbt.com:80 or udp://open.demonii.com:1337/announce), and it doesn’t really matter which one you use. (In fact, the more you use, the easier it will be for others to find you.) For now, suffice it to say that a tracker’s job is to keep other users in the swarm updated about where everyone else is, in case things change and in order to help you find one another in the first place. Obviously, hiding our real identity from trackers as well as the rest of the swarm while still being able to find and share files with one another is a key part of what staying anonymous while using BitTorrent is all about. We’ll look at ways to do that in the next section. [↩]
Another useful consideration for a proxy server I don’t list explicitly is its physical location. You may want to use a proxy located in another country than you are in so as to avoid running afoul of local laws, or to route around geographic censorship. Remember, political and legal borders are not real, so the fundamental Internet technology we’re talking about doesn’t and, to work correctly, shouldn’t pay any attention to them. Most proxy lists display the country a proxy is in alongside its other information. [↩]
Technically, the Tor Browser is a package deal that comes with a modified version of Mozilla Firefox and the actual tor proxy software, all pre-configured to work together. If you want to learn more about Tor and, trust me, you do, consider reading my article all about it. [↩]
Even though each BitTorrent client’s preferences window looks different, they all describe the same basic behavior. [↩]
You can often still download many pieces of a torrent even if there are no seeders in the swarm. Sometimes, if a torrent has many independent files, you can still download the majority of the content you’re after because only a few pieces in a few files are missing. The quickest way to check this is to look for the “Availability” proportion in your BitTorrent client. Without a seed, this readout will be somewhere between 0, meaning absolutely no parts are available, and just under 1. The closer this value is to 1, the more pieces of the torrent are available. I once downloaded a torrent of a four-season TV show whose availability was 0.954, and was happy to find two complete seasons had downloaded without any problem at all despite never finding a seed for that torrent. Other torrents provided the missing episodes and all was well. [↩]
First of all, we need to put policing in the context of a bigger ruling-class project of managing and shaping the working class. I said at the beginning that the emergence of workers’ revolt coincided with a breakdown of old methods of constant personal supervision of the workforce. The state stepped in to provide supervision. The cops were part of that effort, but in the North, the state also expanded its programs of poor relief and public schooling.
Police work was integrated with the system of poor relief, as constables worked on registration of the poor and their placement in workhouses. That’s even before the police were professionalized—the constables were sorting out the “deserving poor” from the “undeserving poor.” If people were unemployed and unable to work, constables would direct them toward charity from churches or the city itself. But if folks were able to work, they were judged to be “idlers” and sent off to the horrors of the workhouse.
The system for poor relief made a crucial contribution to the creation of the market for wage labor. The key function of the relief system was to make unemployment so unpleasant and humiliating that people were willing to take ordinary jobs at very low wages just to avoid unemployment. By punishing the poorest people, capitalism creates a low baseline for the wage scale and pulls the whole scale downward.
The police no longer play such a direct role in selecting people for relief, but they do deliver a good deal of the punishment. As we know, lots of police work has to do with making life unpleasant for unemployed people on the street.
The rise of modern policing also coincides with the rise of public education. Public schools accustom children to the discipline of the capitalist workplace, including the submission to strict rules about the proper time to do things. The school reform movement of the 1830s and 40s also aimed to shape the students’ moral character. The effect of this was supposed to be that students would willingly submit to authority, that they would be able to work hard, exercise self-control, and delay gratification.
In fact, the concepts of good citizenship that came out of school reform movement were perfectly aligned with the concepts of criminology that were being invented to categorize people on the street. The police were to focus not just on crime but on criminal types—a method of profiling backed up by supposedly scientific credentials. The “juvenile delinquent,” for example, is a concept that is common to schooling and policing—and has helped to link the two activities in practice.
This ideology of good citizenship was supposed to have a big effect inside the heads of students, encouraging them to think that the problems in society come from the actions of “bad guys.” A key objective of schooling, according to reformer Horace Mann, should be to implant a certain kind of conscience in the students—so that they discipline their own behavior and begin to police themselves. In Mann’s words, the objective was for children to “think of duty rather than of the policeman.”
Needless to say, an analytic scheme for dividing society between good guys and bad guys is perfect for identifying scapegoats, especially racial ones. Such a moralistic scheme was (and is) also a direct competitor to a class-conscious worldview, which identifies society’s basic antagonism as the conflict between exploiters and exploited. Police activity thus goes beyond simple repression—it “teaches” an ideology of good and bad citizenship that dovetails with the lessons of the classroom and the workhouse.
The overall point here is that the invention of the police was part of a broader expansion of state activity to gain control over the day-to-day behavior of the working class. Schooling, poor relief and police work all aimed to shape workers to become useful to—and loyal to—the capitalist class.