{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Managing assignment files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Distributing assignments to students and collecting them can be a logistical nightmare. If you are running nbgrader on a server, some of this pain can be relieved by relying on nbgrader's built-in functionality for releasing and collecting assignments on the instructor's side, and fetching and submitting assignments on the student's side."
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    ".. contents:: Table of Contents\n",
    "   :depth: 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setting up the exchange"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After an assignment has been created using `nbgrader generate_assignment`, the instructor must actually release that assignment to students. If the class is being taught on a single filesystem, then the instructor may use `nbgrader release_assignment` to copy the assignment files to a shared location on the filesystem for students to then download.\n",
    "\n",
    "First, we must specify a few configuration options. To do this, we'll create a `nbgrader_config.py` file that will get automatically loaded when we run `nbgrader`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing nbgrader_config.py\n"
     ]
    }
   ],
   "source": [
    "%%file nbgrader_config.py\n",
    "\n",
    "c = get_config()\n",
    "\n",
    "c.CourseDirectory.course_id = \"example_course\"\n",
    "c.Exchange.root = \"/tmp/exchange\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the config file, we've specified the \"exchange\" directory to be `/tmp/exchange`. This directory must exist before running `nbgrader`, and it *must* be readable and writable by all users, so we'll first create it and configure the appropriate permissions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "\n",
    "# remove existing directory, so we can start fresh for demo purposes\n",
    "rm -rf /tmp/exchange\n",
    "\n",
    "# create the exchange directory, with write permissions for everyone\n",
    "mkdir /tmp/exchange\n",
    "chmod ugo+rw /tmp/exchange"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Releasing assignments"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    ".. seealso::\n",
    "\n",
    "    :doc:`creating_and_grading_assignments`\n",
    "        Details on generating assignments\n",
    "\n",
    "    :doc:`/command_line_tools/nbgrader-release-assignment`\n",
    "        Command line options for ``nbgrader release_assignment``\n",
    "\n",
    "    :doc:`/command_line_tools/nbgrader-list`\n",
    "        Command line options for ``nbgrader list``\n",
    "\n",
    "    :doc:`philosophy`\n",
    "        More details on how the nbgrader hierarchy is structured.\n",
    "\n",
    "    :doc:`/configuration/config_options`\n",
    "        Details on ``nbgrader_config.py``"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### From the formgrader"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the formgrader extension, you may release assignments by clicking on the \"release\" button:\n",
    "\n",
    "![](images/manage_assignments5.png)\n",
    "\n",
    "**Note** that for the \"release\" button to become available, the `course_id` option must be set in `nbgrader_config.py`.\n",
    "Once completed, you will see a pop-up window with log output:\n",
    "\n",
    "![](images/release_assignment.png)\n",
    "\n",
    "If you decide you want to \"un-release\" an assignment, you may do so by clicking again on the \"release\" button (which is now an \"x\"). **However, note that students who have already downloaded the assignment will still have access to their downloaded copy. Unreleasing an assignment only prevents more students from downloading it.**\n",
    "\n",
    "![](images/manage_assignments6.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### From the command line"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have the directory created, we can actually run `nbgrader release_assignment` (and as with the other nbgrader commands for instructors, this must be run from the root of the course directory):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[ReleaseAssignmentApp | INFO] Source: [NB_GRADER_ROOT]/nbgrader/docs/source/user_guide/release/./ps1\n",
      "[ReleaseAssignmentApp | INFO] Destination: /tmp/exchange/example_course/outbound/ps1\n",
      "[ReleaseAssignmentApp | INFO] Released as: example_course ps1\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "\n",
    "nbgrader release_assignment \"ps1\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, you can verify that the assignment has been appropriately released by running the `nbgrader list` command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[ListApp | INFO] Released assignments:\n",
      "[ListApp | INFO] example_course ps1\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "\n",
    "nbgrader list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that there should only ever be *one* instructor who runs the `nbgrader release_assignment` and `nbgrader collect` commands (and there should probably only be one instructor -- the same instructor -- who runs `nbgrader generate_assignment`, `nbgrader autograde` and the formgrader as well). However this does not mean that only one instructor can do the grading, it just means that only one instructor manages the assignment files. Other instructors can still perform grading by accessing the notebook where the formgrader is running."
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    ".. _fetching-assignments:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fetching assignments"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    ".. seealso::\n",
    "\n",
    "    :doc:`/command_line_tools/nbgrader-fetch-assignment`\n",
    "        Command line options for ``nbgrader fetch_assignment``\n",
    "\n",
    "    :doc:`/command_line_tools/nbgrader-list`\n",
    "        Command line options for ``nbgrader list``\n",
    "\n",
    "    :doc:`/configuration/config_options`\n",
    "        Details on ``nbgrader_config.py``"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the student's perspective, they can list what assignments have been released, and then fetch a copy of the assignment to work on. First, we'll create a temporary directory to represent the student's home directory:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "\n",
    "# remove the fake student home directory if it exists, for demo purposes\n",
    "rm -rf /tmp/student_home\n",
    "\n",
    "# create the fake student home directory and switch to it\n",
    "mkdir /tmp/student_home"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you are not using the default exchange directory (as is the case here), you will additionally need to provide your students with a configuration file that sets the appropriate directory for them:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing /tmp/student_home/nbgrader_config.py\n"
     ]
    }
   ],
   "source": [
    "%%file /tmp/student_home/nbgrader_config.py\n",
    "\n",
    "c = get_config()\n",
    "c.Exchange.root = '/tmp/exchange'\n",
    "c.CourseDirectory.course_id = \"example_course\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### From the notebook dashboard"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    ".. warning::\n",
    "\n",
    "  The \"Assignment List\" extension is not fully compatible with multiple\n",
    "  courses on the same server. Please see :ref:`multiple-classes` for details.\n",
    "\n",
    "Alternatively, students can fetch assignments using the assignment list notebook server extension. You must have installed the extension by following the instructions :doc:`here </user_guide/installation>`, after which you should see an \"Assignments\" tab in dashboard:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](images/assignment_list_released.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The image above shows that there has been one assignment released (\"ps1\") for the class \"example_course\". To get this assignment, students can click the \"Fetch\" button (analogous to running `nbgrader fetch_assignment ps1 --course example_course`. **Note: this assumes nbgrader is always run from the root of the notebook server, which on JupyterHub is most likely the root of the user's home directory.**\n",
    "\n",
    "After the assignment is fetched, it will appear in the list of \"Downloaded assignments\":\n",
    "\n",
    "![](images/assignment_list_downloaded.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Students can click on the name of the assignment to expand it and see all the notebooks in the assignment:\n",
    "\n",
    "![](images/assignment_list_downloaded_expanded.png)\n",
    "\n",
    "Clicking on a particular notebook will open it in a new tab in the browser."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### From the command line"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the student's perspective, they can see what assignments have been released using `nbgrader list`, and passing the name of the class:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[ListApp | INFO] Released assignments:\n",
      "[ListApp | INFO] example_course ps1\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "export HOME=/tmp/student_home && cd $HOME\n",
    "\n",
    "nbgrader list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "They can then fetch an assignment for that class using `nbgrader fetch_assignment` and passing the name of the class and the name of the assignment:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[FetchAssignmentApp | INFO] Source: /tmp/exchange/example_course/outbound/ps1\n",
      "[FetchAssignmentApp | INFO] Destination: /tmp/student_home/ps1\n",
      "[FetchAssignmentApp | INFO] Fetched as: example_course ps1\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "export HOME=/tmp/student_home && cd $HOME\n",
    "\n",
    "nbgrader fetch_assignment \"ps1\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that running `nbgrader fetch_assignment` copies the assignment files from the exchange directory to the local directory, and therefore can be used from any directory:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total ##\n",
      "-rw-r--r-- 1 nb_user nb_group [size] [date] [time] jupyter.png\n",
      "-rw-r--r-- 1 nb_user nb_group [size] [date] [time] problem1.ipynb\n",
      "-rw-r--r-- 1 nb_user nb_group [size] [date] [time] problem2.ipynb\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "\n",
    "ls -l \"/tmp/student_home/ps1\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Additionally, the `nbgrader fetch_assignment` (as well as `nbgrader submit`) command also does not rely on having access to the nbgrader database -- the database is only used by instructors."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Submitting assignments"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    ".. seealso::\n",
    "\n",
    "    :doc:`/command_line_tools/nbgrader-submit`\n",
    "        Command line options for ``nbgrader submit``\n",
    "\n",
    "    :doc:`/command_line_tools/nbgrader-list`\n",
    "        Command line options for ``nbgrader list``\n",
    "\n",
    "    :doc:`/configuration/config_options`\n",
    "        Details on ``nbgrader_config.py``"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### From the notebook dashboard"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    ".. warning::\n",
    "\n",
    "  The \"Assignment List\" extension is not fully compatible with multiple\n",
    "  courses on the same server. Please see :ref:`multiple-classes` for details.\n",
    "\n",
    "Alternatively, students can submit assignments using the assignment list notebook server extension. You must have installed the extension by following the instructions `here <https://github.com/jupyter/nbgrader>`__. Students must have also downloaded the assignments (see :ref:`fetching-assignments`)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After students have worked on the assignment for a while, but before submitting, they can validate that their notebooks pass the tests by clicking the \"Validate\" button (analogous to running `nbgrader validate`). If any tests fail, they will see a warning:\n",
    "\n",
    "![](images/assignment_list_validate_failed.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If there are no errors, they will see that the validation passes:\n",
    "\n",
    "![](images/assignment_list_validate_succeeded.png)"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    ".. note::\n",
    "\n",
    "   If the notebook has been released with hidden tests removed from the source version\n",
    "   (see :ref:`autograder-tests-cell-hidden-tests`) then this validation is only done against the tests the students can\n",
    "   see in the release version."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once students have validated all the notebooks, they can click the \"Submit\" button to submit the assignment (analogous to running `nbgrader submit ps1 --course example_course`). Afterwards, it will show up in the list of submitted assignments (and also still in the list of downloaded assignments):\n",
    "\n",
    "![](images/assignment_list_submitted.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Students may submit an assignment as many times as they'd like. All copies of a submission will show up in the submitted assignments list, and when the instructor collects the assignments, they will get the most recent version of the assignment:\n",
    "\n",
    "![](images/assignment_list_submitted_again.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similarly, if the ``strict`` option (in the student's ``nbgrader_config.py`` file) is set to ``True``, the students will not be able to submit an assignment with missing notebooks (for a given assignment):\n",
    "\n",
    "![](images/assignment_list_submit_error.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### From the command line"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, as a reminder, here is what the student's `nbgrader_config.py` file looks like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "c = get_config()\n",
      "c.Exchange.root = '/tmp/exchange'\n",
      "c.CourseDirectory.course_id = \"example_course\"\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "\n",
    "cat /tmp/student_home/nbgrader_config.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After working on an assignment, the student can submit their version for grading using `nbgrader submit` and passing the name of the assignment and the name of the class:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[SubmitApp | INFO] Source: /tmp/student_home/ps1\n",
      "[SubmitApp | INFO] Destination: /tmp/exchange/example_course/inbound/nb_user+ps1+[timestamp] UTC+[random string]\n",
      "[SubmitApp | INFO] Submitted as: example_course ps1 [timestamp] UTC\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "export HOME=/tmp/student_home && cd $HOME\n",
    "\n",
    "nbgrader submit \"ps1\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that \"the name of the assignment\" really corresponds to \"the name of a folder\". It just happens that, in our current directory, there is a folder called \"ps1\":"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total ##\n",
      "-rw-r--r-- 1 nb_user nb_group [size] [date] [time] nbgrader_config.py\n",
      "drwxr-xr-x 1 nb_user nb_group [size] [date] [time] ps1\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "export HOME=/tmp/student_home && cd $HOME\n",
    "\n",
    "ls -l \"/tmp/student_home\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Students can see what assignments they have submitted using `nbgrader list --inbound`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[ListApp | INFO] Submitted assignments:\n",
      "[ListApp | INFO] example_course nb_user ps1 [timestamp] UTC (no feedback available)\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "export HOME=/tmp/student_home && cd $HOME\n",
    "\n",
    "nbgrader list --inbound"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Importantly, students can run `nbgrader submit` as many times as they want, and all submitted copies of the assignment will be preserved:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[SubmitApp | INFO] Source: /tmp/student_home/ps1\n",
      "[SubmitApp | INFO] Destination: /tmp/exchange/example_course/inbound/nb_user+ps1+[timestamp] UTC+[random string]\n",
      "[SubmitApp | INFO] Submitted as: example_course ps1 [timestamp] UTC\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "export HOME=/tmp/student_home && cd $HOME\n",
    "\n",
    "nbgrader submit \"ps1\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see all versions that have been submitted by again running `nbgrader list --inbound`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[ListApp | INFO] Submitted assignments:\n",
      "[ListApp | INFO] example_course nb_user ps1 [timestamp] UTC (no feedback available)\n",
      "[ListApp | INFO] example_course nb_user ps1 [timestamp] UTC (no feedback available)\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "export HOME=/tmp/student_home && cd $HOME\n",
    "\n",
    "nbgrader list --inbound"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the `nbgrader submit` (as well as `nbgrader fetch_assignment`) command also does not rely on having access to the nbgrader database -- the database is only used by instructors."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "``nbgrader`` requires that the submitted notebook names match the released notebook names for each assignment. For example if a student were to rename one of the given assignment notebooks:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[SubmitApp | INFO] Source: /tmp/student_home/ps1\n",
      "[SubmitApp | INFO] Destination: /tmp/exchange/example_course/inbound/nb_user+ps1+[timestamp] UTC+[random string]\n",
      "[SubmitApp | WARNING] Possible missing notebooks and/or extra notebooks submitted for assignment ps1:\n",
      "    Expected:\n",
      "    \tproblem1.ipynb: MISSING\n",
      "    \tproblem2.ipynb: FOUND\n",
      "    Submitted:\n",
      "    \tmyproblem1.ipynb: EXTRA\n",
      "    \tproblem2.ipynb: OK\n",
      "[SubmitApp | INFO] Submitted as: example_course ps1 [timestamp] UTC\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "export HOME=/tmp/student_home && cd $HOME\n",
    "\n",
    "# assume the student renamed the assignment file\n",
    "mv ps1/problem1.ipynb ps1/myproblem1.ipynb\n",
    "\n",
    "nbgrader submit \"ps1\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By default this assignment will still be submitted however only the \"FOUND\" notebooks (for the given assignment) can be ``autograded`` and will appear on the ``formgrade`` extension. \"EXTRA\" notebooks will not be ``autograded`` and will not appear on the ``formgrade`` extension."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To ensure that students cannot submit an assignment with missing notebooks (for a given assignment) the ``strict`` option, in the student's ``nbgrader_config.py`` file, can be set to ``True``:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting /tmp/student_home/nbgrader_config.py\n"
     ]
    }
   ],
   "source": [
    "%%file /tmp/student_home/nbgrader_config.py\n",
    "\n",
    "c = get_config()\n",
    "c.Exchange.root = '/tmp/exchange'\n",
    "c.CourseDirectory.course_id = \"example_course\"\n",
    "c.ExchangeSubmit.strict = True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "tags": [
     "raises-exception"
    ]
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[SubmitApp | INFO] Source: /tmp/student_home/ps1\n",
      "[SubmitApp | INFO] Destination: /tmp/exchange/example_course/inbound/nb_user+ps1+[timestamp] UTC+[random string]\n",
      "[SubmitApp | CRITICAL] Assignment ps1 not submitted. There are missing notebooks for the submission:\n",
      "    Expected:\n",
      "    \tproblem1.ipynb: MISSING\n",
      "    \tproblem2.ipynb: FOUND\n",
      "    Submitted:\n",
      "    \tmyproblem1.ipynb: EXTRA\n",
      "    \tproblem2.ipynb: OK\n",
      "[SubmitApp | ERROR] nbgrader submit failed\n"
     ]
    },
    {
     "ename": "CalledProcessError",
     "evalue": "Command 'b'export HOME=/tmp/student_home && cd $HOME\\n\\nnbgrader submit \"ps1\"\\n'' returned non-zero exit status 1.",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mCalledProcessError\u001b[0m                        Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-18-63232b70338f>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mget_ipython\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrun_cell_magic\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'bash'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m''\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'export HOME=/tmp/student_home && cd $HOME\\n\\nnbgrader submit \"ps1\"\\n'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m~/miniconda3/envs/nbgrader/lib/python3.9/site-packages/IPython/core/interactiveshell.py\u001b[0m in \u001b[0;36mrun_cell_magic\u001b[0;34m(self, magic_name, line, cell)\u001b[0m\n\u001b[1;32m   2397\u001b[0m             \u001b[0;32mwith\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbuiltin_trap\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2398\u001b[0m                 \u001b[0margs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mmagic_arg_s\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcell\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2399\u001b[0;31m                 \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   2400\u001b[0m             \u001b[0;32mreturn\u001b[0m \u001b[0mresult\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2401\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/miniconda3/envs/nbgrader/lib/python3.9/site-packages/IPython/core/magics/script.py\u001b[0m in \u001b[0;36mnamed_script_magic\u001b[0;34m(line, cell)\u001b[0m\n\u001b[1;32m    140\u001b[0m             \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    141\u001b[0m                 \u001b[0mline\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mscript\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 142\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshebang\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mline\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcell\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    143\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    144\u001b[0m         \u001b[0;31m# write a basic docstring:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<decorator-gen-110>\u001b[0m in \u001b[0;36mshebang\u001b[0;34m(self, line, cell)\u001b[0m\n",
      "\u001b[0;32m~/miniconda3/envs/nbgrader/lib/python3.9/site-packages/IPython/core/magic.py\u001b[0m in \u001b[0;36m<lambda>\u001b[0;34m(f, *a, **k)\u001b[0m\n\u001b[1;32m    185\u001b[0m     \u001b[0;31m# but it's overkill for just that one bit of state.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    186\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mmagic_deco\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0marg\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 187\u001b[0;31m         \u001b[0mcall\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mlambda\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0ma\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mk\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0ma\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mk\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    188\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    189\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mcallable\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0marg\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/miniconda3/envs/nbgrader/lib/python3.9/site-packages/IPython/core/magics/script.py\u001b[0m in \u001b[0;36mshebang\u001b[0;34m(self, line, cell)\u001b[0m\n\u001b[1;32m    243\u001b[0m             \u001b[0msys\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstderr\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mflush\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    244\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mraise_error\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mreturncode\u001b[0m\u001b[0;34m!=\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 245\u001b[0;31m             \u001b[0;32mraise\u001b[0m \u001b[0mCalledProcessError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mreturncode\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcell\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0moutput\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mout\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstderr\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0merr\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    246\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    247\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_run_script\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mp\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcell\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mto_close\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mCalledProcessError\u001b[0m: Command 'b'export HOME=/tmp/student_home && cd $HOME\\n\\nnbgrader submit \"ps1\"\\n'' returned non-zero exit status 1."
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "export HOME=/tmp/student_home && cd $HOME\n",
    "\n",
    "nbgrader submit \"ps1\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Collecting assignments"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    ".. seealso::\n",
    "\n",
    "    :doc:`creating_and_grading_assignments`\n",
    "        Details on grading assignments after they have been collected\n",
    "\n",
    "    :doc:`/command_line_tools/nbgrader-collect`\n",
    "        Command line options for ``nbgrader collect``\n",
    "\n",
    "    :doc:`/command_line_tools/nbgrader-list`\n",
    "        Command line options for ``nbgrader list``\n",
    "\n",
    "    :doc:`philosophy`\n",
    "        More details on how the nbgrader hierarchy is structured.\n",
    "\n",
    "    :doc:`/configuration/config_options`\n",
    "        Details on ``nbgrader_config.py``"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, as a reminder, here is what the instructor's `nbgrader_config.py` file looks like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "\n",
    "cat nbgrader_config.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### From the formgrader"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the formgrader extension, we can collect submissions by clicking on the \"collect\" button:\n",
    "\n",
    "![](images/manage_assignments7.png)\n",
    "\n",
    "As with releasing, this will display a pop-up window when the operation is complete, telling you how many submissions were collected:\n",
    "\n",
    "![](images/collect_assignment.png)\n",
    "\n",
    "From here, you can click on the number of submissions to grade the collected submissions:\n",
    "\n",
    "![](images/manage_assignments8.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### From the command line"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After students have submitted their assignments, the instructor can view what has been submitted with `nbgrader list --inbound`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "\n",
    "nbgrader list --inbound"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The instructor can then collect all submitted assignments with `nbgrader collect` and passing the name of the assignment (and as with the other nbgrader commands for instructors, this must be run from the root of the course directory):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "\n",
    "nbgrader collect \"ps1\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This will copy the student submissions to the `submitted` folder in a way that is automatically compatible with `nbgrader autograde`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "\n",
    "ls -l submitted"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that there should only ever be *one* instructor who runs the `nbgrader release_assignment` and `nbgrader collect` commands (and there should probably only be one instructor -- the same instructor -- who runs `nbgrader generate_assignment`, `nbgrader autograde` and the formgrader as well). However this does not mean that only one instructor can do the grading, it just means that only one instructor manages the assignment files. Other instructors can still perform grading by accessing the notebook server running the formgrader."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python",
   "language": "python",
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
