# Purpose
-KarmaNotes is an online database of college lecture notes. KarmaNotes empowers college students to participate in the free exchange of knowledge.
+KarmaWorld is an online database of college lecture notes. KarmaWorld
+empowers college students to participate in the free exchange of knowledge.
+
+# Naming
+
+The repository and the project are called KarmaWorld. One implementation
+of KarmaWorld, which is run by FinalsClub Foundation, is called
+[KarmaNotes](https://www.karmanotes.org/).
# Pre-Installation
`fabfile.py` (`{project_root}/fabfile.py`) and `README.md`
(`{project_root}/README.md`).
+## External Software Dependencies
+
+### pdf2htmlEX
+
+KarmaWorld uses [pdf2htmlEX](https://github.com/coolwanglu/pdf2htmlEX) as
+a dependency. pdf2htmlEX is used to convert uploaded PDF notes into HTML.
+
+An [outdated version of pdf2htmlEX](https://github.com/FinalsClub/pdf2htmlEX)
+is available which includes the
+[patch](https://github.com/FinalsClub/pdf2htmlEX/commit/3c19f6abd8d59d1b58cf254b7160b332b3f5b517)
+required for pdf2htmlEX to correctly work with KarmaWorld.
+
+Newer versions can be used by applying the patch by hand. It's a fairly
+simple two-line modification that can be done after installing
+pdf2htmlEX.
+
+### SSL Certificate
+
+If you wish to host your system publicly, you'll almost certainly want
+an SSL certificate signed by a proper authority.
+
+You may need to set the `SSL_REDIRECT` environment variable to `true` to
+make KarmaWorld redirect insecure connections to secure ones.
+
+Follow [Heroku's SSL setup](https://devcenter.heroku.com/articles/ssl-endpoint)
+to get SSL running on your server with Heroku.
+
## External Service Dependencies
Notice: This software makes use of external third party services which require
accounts to access the service APIs. Without these third parties available,
this software may require considerable overhaul. These services have
-API keys, credentials, and other information that you must provide to KarmaNotes
+API keys, credentials, and other information that you must provide to KarmaWorld
as environment variables. The best way to persist these environment variables is
-by using a `.env` file. Copy `.env.example` to `.env` and populate the fields as
-required.
+by using a `.env` file. Copy `.env.example` to `.env` and populate the fields as required.
+
+A number of services are required even if running the KarmaWorld web service
+locally, some of the services are recommended, and some are completely optional
+even if running the web service on Heroku.
+
+Many of these services have free tiers and can be used without charge for
+development testing purposes.
+
+* Reminder
+ * Copy `.env.example` to `.env` and populate the environment variables there.
+* Required Services
+ * [Google Drive](#google-drive)
+ * [Filepicker](#filepicker)
+ * [PostgreSQL](#postgresql)
+ * [Celery](#celery-queue)
+* Optional but recommended
+ * [IndexDen](#indexden): enables searching through courses, notes, etc
+ * [Heroku](#heroku): the production environment used by karmanotes.org
+ * it might not be possible to run KarmaWorld on Heroku using a free
+ webapp.
+ * [Amazon S3](#s3-for-static-files): for static file hosting
+* Entirely optional (though used in the production environment)
+ * [Twitter](#twitter): share updates about new uploads
+ * [Amazon Mechanical Turk](#amazon-mechanical-turk): generate quizzes, flashcards, etc
+ * [Amazon CloudFront](#amazon-cloudfront-cdn)
+ * [Amazon S3](#s3-for-filepicker): store files uploaded to Filepicker
+ * Filepicker does not support S3 storage in its free tier
### Heroku
This project has chosen to use [Heroku](www.heroku.com) to host the Django and
See README.heroku for more information.
+#### pdf2htmlEX on Heroku
+If using Heroku, the default
+[KarmaNotes Heroku buildpack](https://github.com/FinalsClub/heroku-buildpack-karmanotes)
+will [include](https://github.com/FinalsClub/heroku-buildpack-karmanotes/blob/master/bin/steps/pdf2htmlex)
+the [required version of pdf2htmlEX](#pdf2htmlex).
+
+### Celery Queue
+Celery uses the Apache Message Queueing Protocol for passing messages to its workers.
+
+For production, we recommend using Heroku's CloudAMQP add-on, getting your own CloudAMQP account, or
+running a queueing system on your own. The `CLOUDAMQP_URL` environment variable must be set correctly
+for KarmaWorld to be able to use Celery. The `CELERY_QUEUE_NAME` environment variable
+must be set to the name of the queue you wish to use. Settings this to something unique
+allows multiple instances of KarmaWorld (or some other software) to share the same queueing server.
+
+For development on localhost, `RabbitMQ` is the default for `djcelery` and is well supported. Ensure
+`RabbitMQ` is installed for local development.
+
+### PostgreSQL
+
+PostgreSQL is not necessarily required; other RDBMS could probably be fit into
+place. However, the code was largely written assuming PostgreSQL will be used.
+Change to another system with the caveat that it might take some work.
+
+There are many cloud providers which provide PostgreSQL databases. Heroku has
+an add-on for providing a PostgreSQL database. Ensure something like this
+is made available and installed to the app.
+
+For local development, ensure a PostgreSQL is running on localhost or is
+otherwise accessible.
+
### Amazon S3
The instructions for creating an [S3](http://aws.amazon.com/s3/) bucket may be
[found on Amazon.](http://docs.aws.amazon.com/AmazonS3/latest/gsg/CreatingABucket.html)
-Two, separate buckets will be needed in production: one for static file hosting
+Two, separate buckets may be used in production: one for static file hosting
and one as a communication bus with Filepicker.
+#### S3 for Filepicker
+
This software uses S3 to store files which are sent to or received
from Filepicker. Filepicker will need to know the S3 bucket name, access key,
and secret key.
bucket, because the software will upload files to the Filepicker S3 bucket
through Filepicker's API and it will link to or download files from the
Filepicker S3 bucket through Filepicker's URLs. This will be covered in the
-Filepicker section below.
+[Filepicker section](#filepicker).
+
+#### S3 for static files
This software uses S3 for hosting static files. The software will need to
update static files on the S3 bucket. As such, the software will need the
supply the credentials for the user with access to the Filepicker S3 bucket.
### IndexDen
-KarmaNotes uses IndexDen to create a searchable index of all the notes in the
+KarmaWorld uses IndexDen to create a searchable index of all the notes in the
system. Create an free IndexDen account at
[their homepage](http://indexden.com/). You will be given a private URL that
accesses your IndexDen account. This URL is visible on your dashboard (you
Set the `INDEXDEN_PRIVATE_URL` environment variable to your private URL.
Set the `INDEXDEN_INDEX` environment variable to the name of the index you want
-to use for KarmaNotes. The index will be created automatically when KarmaNotes
+to use for KarmaWorld. The index will be created automatically when KarmaNotes
is run if it doesn't already exist. It may be created through the GUI if
desired.
environment variables `TWITTER_CONSUMER_KEY`, `TWITTER_CONSUMER_SECRET`,
`TWITTER_ACCESS_TOKEN_KEY`, `TWITTER_ACCESS_TOKEN_SECRET`.
-### SSL Certificate
+# Local
-If you wish to host your system publicly, you'll need an SSL certificate
-signed by a proper authority.
+## Configuring foreman
-Follow [Heroku's SSL setup](https://devcenter.heroku.com/articles/ssl-endpoint)
-to get SSL running on your server.
+KarmaNotes runs on Heroku as a webapp and thus makes use of a Procfie. While
+not strictly necessary, KarmaWorld can use the same basic Procfile which is
+convenient and consistent.
-You may set the `SSL_REDIRECT` environment variable to `true` to make KarmaNotes
-redirect insecure connections to secure ones.
+To use the Procfile locally, we recommend using `foreman`. To install `foreman`
+and other Heroku tools, install the
+[Heroku toolbelt](https://toolbelt.heroku.com/).
-# Local Install
+Ensure environment variables are available to `foreman` by copying
+`.env.example` to `.env` and update those variables as appropriate for your
+local system.
-KarmaNotes is a Heroku application. Download the [Heroku toolbelt](https://toolbelt.heroku.com/).
+## pdf2htmlEX
+
+This project uses [pdf2htmlEX](https://github.com/coolwanglu/pdf2htmlEX) as
+a dependency. pdf2htmlEX is used to convert uploaded PDF notes into HTML. It
+needs to be installed on the same system that KarmaWorld is running on.
+
+### using their source
+
+See their instructions at
+[https://github.com/coolwanglu/pdf2htmlEX/wiki/Building](https://github.com/coolwanglu/pdf2htmlEX/wiki/Building).
+
+Make sure to [patch](https://github.com/FinalsClub/pdf2htmlEX/commit/3c19f6abd8d59d1b58cf254b7160b332b3f5b517)
+the source code to expose two variables.
+
+### using our fork
+
+You can use FinalsClub's [outdated version of pdf2htmlEX](https://github.com/FinalsClub/pdf2htmlEX).
+See their installation instructions above, but don't worry about patching.
+
+### using their PPA
+
+You can use [their upstream PPA](https://launchpad.net/~coolwanglu/+archive/ubuntu/pdf2htmlex).
+
+ apt-add-repository ppa:coolwanglu/pdf2htmlex
+ apt-get update
+ apt-get install pdf2htmlex
+
+Then patch the javascript on your system by running this code in the shell.
+
+ cat >> `dpkg -L pdf2htmlex | grep pdf2htmlEX.js` <<PDF2HTMLEXHACK
+ Viewer.prototype['rescale'] = Viewer.prototype.rescale;
+ Viewer.prototype['scroll_to'] = Viewer.prototype.scroll_to;
+ PDF2HTMLEXHACK
+
+## Install
-Before your running it for the first time, there are
-a few setup steps:
1. `virtualenv venv`
1. `source venv/bin/activate`
1. `pip install -r requirements.txt`
+ * on Debian systems, some packages are required for pip to succeed:
+ * `apt-get install python-dev libpython-dev python-psycopg2 libmemcached-dev libffi-dev libssl-dev postgresql-server-dev-X.Y`
1. `pip install -r requirements-dev.txt`
+
+## Configuration
+
+Make sure [External Service Dependencies](#external_service_dependencies) are
+satisfied. This includes running a local database and RabbitMQ instance as
+desired.
+
+ 1. configure `.env` as per [instructions](#external_service_dependencies)
1. `foreman run python manage.py syncdb --migrate --noinput`
1. `foreman run python manage.py createsuperuser`
1. `foreman run python manage.py fetch_usde_csv ./schools.csv`
- 1. `foreman run python manage.py import_usde _csv ./schools.csv`
+ 1. `foreman run python manage.py import_usde_csv ./schools.csv`
1. `foreman run python manage.py sanitize_usde_schools`
-To run KarmaNotes locally, make sure you are inside your
-virtual environment (`source venv/bin/activate`) and run `foreman start`.
+* `fetch_usde_csv` downloads school records and stores them to `./schools.csv`. This file name
+ and location is arbitrary. As long as the same file is passed into `import_usde_csv` for
+ reading, everything should be fine.
+
+* `fetching_usde_csv` requires `7zip` to be installed for processing compressed
+ archives. On Debian-based systems, this entails `apt-get install p7zip-full`
+
+## Run
+
+Make sure you are inside your virtual environment (`source venv/bin/activate`).
+
+If the code has changed or this is the first run, make sure any modified static
+files get compressed with `foreman run python manage.py compress`. Static files
+then need to be uploaded correctly with `foreman run python manage.py
+collectstatic`.
+
+Run `foreman start`. `foreman` will load the `.env` file and manage running all
+processes in a way that is similar to that of Heroku. This allows better
+consistency with local, staging, and production deployments.
+
+To run web-only, but no celery or beat, run `foreman start web` to specify
+strictly the web worker.
+
Press ctrl-C to kill foreman. Foreman will run Django's runserver command.
If you wish to have more control over how this is done, you can do
`foreman run python manage.py runserver <options>`. For running any other
# Heroku Install
-KarmaNotes is a Heroku application. Download the [Heroku toolbelt](https://toolbelt.heroku.com/).
+KarmaNotes runs on Heroku as a webapp. This section addresses what was done
+for KarmaNotes so that other implementations of KarmaWorld can be run on
+Heroku.
+
+Before anything else, download the [Heroku toolbelt](https://toolbelt.heroku.com/).
-To run KarmaNotes on Heroku, do `heroku create` and `git push heroku master` as typical
+To run KarmaWorld on Heroku, do `heroku create` and `git push heroku master` as typical
for a Heroku application. Set your the variable `BUILDPACK_URL` to
-`https://github.com/charlesconnell/heroku-buildpack-karmanotes` to use a buildpack designed
-to support KarmaNotes.
+`https://github.com/FinalsClub/heroku-buildpack-karmanotes` to use a buildpack
+designed to support KarmaNotes.
You will need to import the US Department of Education's list of accredited schools.
1. Fetch USDE schools with
`heroku run python manage.py fetch_usde_csv ./schools.csv`
1. Upload the schools into the database with
- `heroku run python /manage.py import_usde _csv ./schools.csv`
+ `heroku run python /manage.py import_usde_csv ./schools.csv`
1. Clean up redundant information with
`heroku run python /manage.py sanitize_usde_schools`
projects / oweals / karmaworld.git / blobdiff